{"version":"1.0","provider_name":"Kodlar\u0131n Gizemli D\u00fcnyas\u0131","provider_url":"https:\/\/fatihsoysal.com\/blog","author_name":"Fatih Soysal","author_url":"https:\/\/fatihsoysal.com\/blog\/author\/fatihsoysal\/","title":"Do\u011fal Dil \u0130\u015flemede Tokenle\u015ftirme: Kapsaml\u0131 Bir Bak\u0131\u015f","type":"rich","width":600,"height":338,"html":"<blockquote class=\"wp-embedded-content\" data-secret=\"4QQL3Aoo2V\"><a href=\"https:\/\/fatihsoysal.com\/blog\/dogal-dil-islemede-tokenlestirme-kapsamli-bir-bakis\/\">Do\u011fal Dil \u0130\u015flemede Tokenle\u015ftirme: Kapsaml\u0131 Bir Bak\u0131\u015f<\/a><\/blockquote><iframe sandbox=\"allow-scripts\" security=\"restricted\" src=\"https:\/\/fatihsoysal.com\/blog\/dogal-dil-islemede-tokenlestirme-kapsamli-bir-bakis\/embed\/#?secret=4QQL3Aoo2V\" width=\"600\" height=\"338\" title=\"&#8220;Do\u011fal Dil \u0130\u015flemede Tokenle\u015ftirme: Kapsaml\u0131 Bir Bak\u0131\u015f&#8221; &#8212; Kodlar\u0131n Gizemli D\u00fcnyas\u0131\" data-secret=\"4QQL3Aoo2V\" frameborder=\"0\" marginwidth=\"0\" marginheight=\"0\" scrolling=\"no\" class=\"wp-embedded-content\"><\/iframe><script>\n\/*! This file is auto-generated *\/\n!function(d,l){\"use strict\";l.querySelector&&d.addEventListener&&\"undefined\"!=typeof URL&&(d.wp=d.wp||{},d.wp.receiveEmbedMessage||(d.wp.receiveEmbedMessage=function(e){var t=e.data;if((t||t.secret||t.message||t.value)&&!\/[^a-zA-Z0-9]\/.test(t.secret)){for(var s,r,n,a=l.querySelectorAll('iframe[data-secret=\"'+t.secret+'\"]'),o=l.querySelectorAll('blockquote[data-secret=\"'+t.secret+'\"]'),c=new RegExp(\"^https?:$\",\"i\"),i=0;i<o.length;i++)o[i].style.display=\"none\";for(i=0;i<a.length;i++)s=a[i],e.source===s.contentWindow&&(s.removeAttribute(\"style\"),\"height\"===t.message?(1e3<(r=parseInt(t.value,10))?r=1e3:~~r<200&&(r=200),s.height=r):\"link\"===t.message&&(r=new URL(s.getAttribute(\"src\")),n=new URL(t.value),c.test(n.protocol))&&n.host===r.host&&l.activeElement===s&&(d.top.location.href=t.value))}},d.addEventListener(\"message\",d.wp.receiveEmbedMessage,!1),l.addEventListener(\"DOMContentLoaded\",function(){for(var e,t,s=l.querySelectorAll(\"iframe.wp-embedded-content\"),r=0;r<s.length;r++)(t=(e=s[r]).getAttribute(\"data-secret\"))||(t=Math.random().toString(36).substring(2,12),e.src+=\"#?secret=\"+t,e.setAttribute(\"data-secret\",t)),e.contentWindow.postMessage({message:\"ready\",secret:t},\"*\")},!1)))}(window,document);\n<\/script>\n","description":"Do\u011fal dil i\u015fleme (NLP), bilgisayarlar\u0131n insan dilini anlamas\u0131n\u0131 ve i\u015flemesini sa\u011flayan bir alan. NLP'nin temel yap\u0131 ta\u015flar\u0131ndan biri ise tokenle\u015ftirme (tokenization) i\u015flemidir. Basit\u00e7e a\u00e7\u0131klamak gerekirse, tokenle\u015ftirme, metin verilerini anlaml\u0131 birimlere (token'lara) ay\u0131rma i\u015flemidir. Bu birimler genellikle kelimeler, say\u0131lar veya noktalama i\u015faretleri olabilir. Ancak, i\u015fin as\u0131l inceli\u011fi bu token'lar\u0131n nas\u0131l ve hangi kriterlere g\u00f6re ayr\u0131\u015ft\u0131r\u0131laca\u011f\u0131nda yatmaktad\u0131r. \u00d6rne\u011fin, \"elma ve armut\" c\u00fcmlesini ele alal\u0131m. Basit bir tokenle\u015ftirme y\u00f6ntemi bize \"elma\", \"ve\", \"armut\" tokenlar\u0131n\u0131 verecektir. Ancak daha karma\u015f\u0131k durumlar i\u00e7in daha geli\u015fmi\u015f y\u00f6ntemler gereklidir."}