Linux¾µäµÄ¼¸¿îÊÕ°üÒýÇæ
±¾ÎÄö¾ÙËĸö½ÏÁ¿¾µäµÄ Linux ÊÕ°üÒýÇ棬ÈôÊÇÉÐÓÐÆäËûÄãÒÔΪokµÄ¿ÉÒÔÁôÑÔ¡£ÕâËĸö»®·ÖÊÇ£º
libpcap/libpcap-mmap
PF_RING
DPDK
xdp
libpcap
libpcapµÄ°ü²¶»ñ»úÖÆÊÇÔÚÊý¾ÝÁ´Â·²ãÔöÌíÒ»¸öÅÔ·´¦Àí£¬²»×ÌÈÅϵͳ×ÔÉíµÄÍø·ÐÒéÕ»µÄ´¦Àí£¬¶Ô·¢ËͺÍÎüÊÕµÄÊý¾Ý°üͨ¹ýLinuxÄÚºË×ö¹ýÂË»ººÍ³å´¦Àí£¬×îºóÖ±½Óת´ï¸øÉϲãÓ¦ÓóÌÐò¡£
Êý¾Ý°üµÖ´ïÍø¿¨×°±¸¡£
Íø¿¨×°±¸ÒÀ¾ÝÉèÖþÙÐÐDMA²Ù×÷¡££¨ ¡¸µÚ1´Î¿½±´¡¹ £ºÍø¿¨¼Ä´æÆ÷->ÄÚºËΪÍø¿¨·ÖÅɵĻº³åÇøring buffer£©
Íø¿¨·¢ËÍÖÐÖ¹£¬½ÐÐÑ´¦ÀíÆ÷¡£
Çý¶¯Èí¼þ´Óring bufferÖжÁÈ¡£¬Ìî³äÄÚºËskbuff½á¹¹£¨ ¡¸µÚ2´Î¿½±´¡¹ £ºÄÚºËÍø¿¨»º³åÇøring buffer->ÄÚºËרÓÃÊý¾Ý½á¹¹skbuff£©
½Ó×ÅŲÓÃnetif_receive_skbº¯Êý£º
5.1 ÈôÊÇÓÐ×¥°ü³ÌÐò£¬ÓÉÍøÂç·Ö½Ó¿Ú½øÈëBPF¹ýÂËÆ÷£¬½«¹æÔòÆ¥ÅäµÄ±¨ÎÄ¿½±´µ½ÏµÍ³Äں˻º´æ £¨ ¡¸µÚ3´Î¿½±´¡¹ £©¡£BPFΪÿһ¸öÒªÇóЧÀ͵Ä×¥°ü³ÌÐò¹ØÁªÒ»¸öfilterºÍÁ½¸öbuffer¡£BPF·ÖÅÉbuffer ÇÒͨ³£ÇéÐÎÏÂËüµÄ¶î¶ÈÊÇ4KB the store buffer ±»Ê¹ÓÃÀ´ÎüÊÕÀ´×ÔÊÊÅäÆ÷µÄÊý¾Ý£»the hold buffer±»Ê¹ÓÃÀ´¿½±´°üµ½Ó¦ÓóÌÐò¡£
5.2 ´¦ÀíÊý¾ÝÁ´Â·²ãµÄÇŽӹ¦Ð§£»
5.3 ƾ֤skb->protocol×Ö¶ÎÈ·¶¨ÉϲãÐÒé²¢Ìá½»¸øÍøÂç²ã´¦Àí£¬½øÈëÍøÂçÐÒéÕ»£¬¾ÙÐи߲㴦Àí¡£
libpcapÈƹýÁËLinuxÄÚºËÊÕ°üÁ÷³ÌÖÐÐÒéÕ»²¿·ÖµÄ´¦Àí£¬Ê¹µÃÓû§¿Õ¼äAPI¿ÉÒÔÖ±½ÓŲÓÃÌ×½Ó×ÖPF_PACKET´ÓÁ´Â·²ãÇý¶¯³ÌÐòÖлñµÃÊý¾Ý±¨ÎĵĿ½±´£¬½«Æä´ÓÄں˻º³åÇø¿½±´ÖÁÓû§¿Õ¼ä»º³åÇø£¨ ¡¸µÚ4´Î¿½±´¡¹ £©
libpcap-mmap
libpcap-mmapÊǶԾɵÄlibpcapʵÏÖµÄˢУ¬Ð°汾µÄlibpcap»ù±¾¶¼½ÓÄÉpacket_mmap»úÖÆ¡£PACKET_MMAPͨ¹ýmmap£¬ïÔÌÒ»´ÎÄڴ濽±´£¨ ¡¸µÚ4´Î¿½±´Ã»ÓÐÁË¡¹ £©£¬ïÔÌÁËƵÈÔµÄϵͳŲÓ㬴ó´óÌá¸ßÁ˱¨ÎIJ¶»ñµÄЧÂÊ¡£
PF_RING
ÎÒÃÇ¿´µ½Ö®Ç°libpcapÓÐ4´ÎÄڴ濽±´¡£libpcap_mmapÓÐ3´ÎÄڴ濽±´¡£PF_RINGÌá³öµÄ½¹µã½â¾ö¼Æ»®¼´ÊÇïÔ̱¨ÎÄÔÚ´«ÊäÀú³ÌÖеĿ½±´´ÎÊý¡£
ÎÒÃÇ¿ÉÒÔ¿´µ½£¬Ïà¶ÔÓëlibpcap_mmapÀ´Ëµ£¬pfringÔÊÐíÓû§¿Õ¼äÄÚ´æÖ±½ÓºÍrx_buffer×ömmap¡£ÕâÓÖïÔÌÁËÒ»´Î¿½±´ £¨ ¡¸libpcap_mmapµÄµÚ2´Î¿½±´¡¹£ºrx_buffer->skb£©
PF-RING ZCʵÏÖÁËDNA£¨Direct NIC Access Ö±½ÓÍø¿¨»á¼û£©ÊÖÒÕ£¬½«Óû§ÄÚ´æ¿Õ¼äÓ³Éäµ½Çý¶¯µÄÄÚ´æ¿Õ¼ä£¬Ê¹Óû§µÄÓ¦ÓÿÉÒÔÖ±½Ó»á¼ûÍø¿¨µÄ¼Ä´æÆ÷ºÍÊý¾Ý¡£
ͨ¹ýÕâÑùµÄ·½·¨£¬×èÖ¹ÁËÔÚÄں˶ÔÊý¾Ý°ü»º´æ£¬ïÔÌÁËÒ»´Î¿½±´£¨ ¡¸libpcapµÄµÚ1´Î¿½±´¡¹ £¬DMAµ½Äں˻º³åÇøµÄ¿½±´£©¡£Õâ¾ÍÊÇÍêÈ«µÄÁ㿽±´¡£
ÆäÈõµãÊÇ£¬Ö»ÓÐÒ»¸ö Ó¦ÓÿÉÒÔÔÚij¸öʱ¼ä·¿ªDMA ring£¨Çë×¢ÖØ£¬ÏÖÔÚµÄÍø¿¨¿ÉÒÔ¾ßÓжà¸öRX / TXÐÐÁУ¬´Ó¶ø¾Í¿ÉÒÔÔÚÿ¸öÐÐÁÐÉÏͬʱһ¸öÓ¦ÓóÌÐò£©£¬»»¶øÑÔÖ®£¬Óû§Ì¬µÄ¶à¸öÓ¦ÓÃÐèÒªÏàÏ໥ͬ²Å»ª·Ö·¢Êý¾Ý°ü¡£
DPDK
pf-ring zcºÍdpdk¾ù¿ÉÒÔʵÏÖÊý¾Ý°üµÄÁ㿽±´£¬Á½Õß¾ùÅÔ·ÁËÄںˣ¬¿ÉÊÇʵÏÖÔÀíÂÔÓвî±ð¡£pf-ring zcͨ¹ýzcÇý¶¯£¨Ò²ÔÚÓ¦Óò㣩½ÓÊÜÊý¾Ý°ü£¬dpdk»ùÓÚUIOʵÏÖ¡£
1 UIO+mmap ʵÏÖÁ㿽±´£¨zero copy£©
UIO£¨Userspace I/O£©ÊÇÔËÐÐÔÚÓû§¿Õ¼äµÄI/OÊÖÒÕ¡£LinuxϵͳÖÐÒ»Ñùƽ³£µÄÇý¶¯×°±¸¶¼ÊÇÔËÐÐÔÚÄں˿ռ䣬¶øÔÚÓû§¿Õ¼äÓÃÓ¦ÓóÌÐòŲÓü´¿É£¬¶øUIOÔòÊǽ«Çý¶¯µÄºÜÉÙÒ»²¿·ÖÔËÐÐÔÚÄں˿ռ䣬¶øÔÚÓû§¿Õ¼äʵÏÖÇý¶¯µÄ¾ø´ó´ó¶¼¹¦Ð§¡£½ÓÄÉLinuxÌṩUIO»úÖÆ£¬¿ÉÒÔÅÔ·Kernel£¬½«ËùÓб¨ÎÄ´¦ÀíµÄÊÂÇéÔÚÓû§¿Õ¼äÍê³É¡£
2 UIO+PMD ïÔÌÖÐÖ¹ºÍCPUÉÏÏÂÎÄÇл»
DPDKµÄUIOÇý¶¯ÆÁÕÏÁËÓ²¼þ·¢³öÖÐÖ¹£¬È»ºóÔÚÓû§Ì¬½ÓÄÉ×Ô¶¯ÂÖѯµÄ·½·¨£¬ÕâÖÖģʽ±»³ÆΪPMD£¨Poll Mode Driver£©¡£
ÓëDPDKÏà±È£¬pf-ring£¨no zc£©Ê¹ÓõÄÊÇNAPI pollingºÍÓ¦Óòãpolling£¬¶øpf-ring zcÓëDPDKÀàËÆ£¬½öʹÓÃÓ¦Óòãpolling¡£
3 HugePages ïÔÌTLB miss
ÔÚ²Ù×÷ϵͳÒýÈëMMU£¨Memory Management Unit£©ºó£¬CPU¶ÁÈ¡ÄÚ´æµÄÊý¾ÝÐèÒªÁ½´Î»á¼ûÄÚ´æ¡£µÚÒ»´ÎÒªÅÌÎÊÒ³±í½«Âß¼µØµãת»»ÎªÎïÀíµØµã£¬È»ºó»á¼û¸ÃÎïÀíµØµã¶ÁÈ¡Êý¾Ý»òÖ¸Áî¡£
ΪÁËïÔÌÒ³Êý¹ý¶à£¬Ò³±í¹ý´ó¶øµ¼ÖµÄÅÌÎÊʱ¼ä¹ý³¤µÄÎÊÌ⣬±ãÒýÈëÁËTLB(Translation Lookaside Buffer)£¬¿É·ÒëΪµØµãת»»»º³åÆ÷¡£TLBÊÇÒ»¸öÄÚ´æÖÎÀíµ¥Î»£¬Ò»Ñùƽ³£´æ´¢ÔڼĴæÆ÷ÖУ¬ÄÚÀï´æ´¢ÁËÄ¿½ñ×î¿ÉÄܱ»»á¼ûµ½µÄһС²¿·ÖÒ³±íÏî¡£
ÒýÈëTLBºó£¬CPU»áÊ×ÏÈÈ¥TLBÖÐÑ°Ö·£¬ÓÉÓÚTLB´æ·ÅÔڼĴæÆ÷ÖУ¬ÇÒÆäÖ»°üÀ¨Ò»Ð¡²¿·ÖÒ³±íÏÒò´ËÅÌÎÊËÙÂʷdz£¿ì¡£ÈôTLBÖÐÑ°Ö·Àֳɣ¨TLB hit£©£¬ÔòÎÞÐèÔÙÈ¥RAMÖÐÅÌÎÊÒ³±í£»ÈôTLBÖÐѰַʧ°Ü£¨TLB miss£©£¬ÔòÐèҪȥRAMÖÐÅÌÎÊÒ³±í£¬ÅÌÎʵ½ºó£¬»á½«¸ÃÒ³¸üÐÂÖÁTLBÖС£
¶øDPDK½ÓÄÉHugePages £¬ÔÚx86-64ÏÂÖ§³Ö2MB¡¢1GBµÄÒ³¾Þϸ£¬´ó´ó½µµÍÁË×ÜÒ³¸öÊýºÍÒ³±íµÄ¾Þϸ£¬´Ó¶ø´ó´ó½µµÍTLB missµÄ¼¸ÂÊ£¬ÌáÉýCPUÑ°Ö·ÐÔÄÜ¡£
4 ÆäËüÓÅ»¯
SNA£¨Shared-nothing Architecture£©£¬Èí¼þ¼Ü¹¹È¥ÖÐÐÄ»¯£¬Ö»¹Ü×èֹȫ¾Ö¹²Ïí£¬´øÀ´È«¾Ö¾ºÕù£¬Ê§È¥ºáÏòÀ©Õ¹µÄÄÜÁ¦¡£NUMAϵͳϲ»¿çNodeÔ¶³ÌʹÓÃÄÚ´æ¡£
SIMD£¨Single Instruction Multiple Data£©£¬´Ó×îÔçµÄmmx/sseµ½×îеÄavx2£¬SIMDµÄÄÜÁ¦Ò»Ö±ÔÚÔöÇ¿¡£DPDK½ÓÄÉÅúÁ¿Í¬Ê±´¦Àí¶à¸ö°ü£¬ÔÙÓÃÏòÁ¿±à³Ì£¬Ò»¸öÖÜÆÚÄÚ¶ÔËùÓаü¾ÙÐд¦Àí¡£ºÃ±È£¬memcpy¾ÍʹÓÃSIMDÀ´Ìá¸ßËÙÂÊ¡£
cpu affinity£º¼´ CPU Ç׺ÍÐÔ
XDP
xdp´ú±íeXpressÊý¾Ý·¾¶£¬Ê¹ÓÃebpf ×ö°ü¹ýÂË£¬Ïà¹ØÓÚdpdk½«Êý¾Ý°üÖ±½ÓË͵½Óû§Ì¬£¬ÓÃÓû§Ì¬µ±×ö¿ìËÙÊý¾Ý´¦ÀíƽÃ棬xdpÊÇÔÚÇý¶¯²ã½¨ÉèÁËÒ»¸öÊý¾Ý¿ìËÙƽÃæ¡£ÔÚÊý¾Ý±»Íø¿¨Ó²¼þdmaµ½Äڴ棬·ÖÅÉskb֮ǰ£¬¶ÔÊý¾Ý°ü¾ÙÐд¦Àí¡£
Çë×¢ÖØ£¬XDP²¢Ã»ÓжÔÊý¾Ý°ü×öKernel bypass£¬ËüÖ»ÊÇÌáÇ°×öÁËÒ»µãÔ¤¼ì°ÕÁË¡£
Ïà¹ØÓÚDPDK£¬XDP¾ßÓÐÒÔÏÂÓŵ㣺
ÎÞÐèµÚÈý·½´úÂë¿âºÍÔÊÐí
ͬʱ֧³ÖÂÖѯʽºÍÖÐֹʽÍøÂç
ÎÞÐè·ÖÅÉ´óÒ³
ÎÞÐèרÓõÄCPU
ÎÞÐè½ç˵еÄÇå¾²ÍøÂçÄ£×Ó
XDPµÄʹÓó¡¾°°üÀ¨£º
DDoS·ÀÓù
·À»ðǽ
»ùÓÚXDP_TXµÄ¸ºÔØƽºâ
ÍøÂçͳ¼Æ
ÖØ´óÍøÂç²ÉÑù
¸ßËÙÉúÒâӪҵƽ̨
OK£¬ÒÔÉϾÍÊǽñÌìµÄ·ÖÏí£¬ÈôÊÇÄãÒÔΪÉÐÓÐÆäËûµÄÊÕ°üÒýÇ棬¿ÉÒÔÁôÑÔ·ÖÏí¡£
ÒÔÉϾÍÊÇLinux¾µäµÄ¼¸¿îÊÕ°üÒýÇæµÄÏêϸÄÚÈÝ£¬¸ü¶àÇë¹Ø×¢±¾ÍøÄÚÆäËüÏà¹ØÎÄÕ£¡