Redis cluster集群Hash Tag原理分析

作者：竹園李 2023-06-10 23:09:40

hash tag是一把雙刃劍，在使用時需要考慮具體業務邏輯與場景，應當盡量避免上述問題。假設無法避免時，可以對key的粒度按照業務線或者場景進行細化，進而對key進行拆分，以便更均勻的分散到不同的slot上。

工欲善其事必先利其器，在正式開始研究redis cluster hash tag之前，我們先以最小的成本搭建一套cluster集群。

docker-compose搭建redis cluster

這里使用dockerhub的bitnami/redis-cluster進行搭建，過程相比網上其他教程而言非常簡單，2個shell命令搞定。

curl -sSL https://raw.githubusercontent.com/bitnami/containers/main/bitnami/redis-cluster/docker-compose.yml > docker-compose.yml
docker-compose up -d

bitnami redis-cluster 對應的 docker-compose.yml (筆者做了細微的修改)

version:'3'
services:
  redis-node-0:
    image:docker.io/bitnami/redis-cluster:7.0
    container_name:node-0
    volumes:
      -redis-cluster_data-0:/bitnami/redis/data
    environment:
      -'REDIS_PASSWORD=123'
      -'REDIS_NODES=redis-node-0 redis-node-1 redis-node-2 redis-node-3 redis-node-4 redis-node-5'

  redis-node-1:
    image:docker.io/bitnami/redis-cluster:7.0
    container_name:node-1
    volumes:
      -redis-cluster_data-1:/bitnami/redis/data
    environment:
      -'REDIS_PASSWORD=123'
      -'REDIS_NODES=redis-node-0 redis-node-1 redis-node-2 redis-node-3 redis-node-4 redis-node-5'

  redis-node-2:
    image:docker.io/bitnami/redis-cluster:7.0
    container_name:node-2
    volumes:
      -redis-cluster_data-2:/bitnami/redis/data
    environment:
      -'REDIS_PASSWORD=123'
      -'REDIS_NODES=redis-node-0 redis-node-1 redis-node-2 redis-node-3 redis-node-4 redis-node-5'

  redis-node-3:
    image:docker.io/bitnami/redis-cluster:7.0
    container_name:node-3
    volumes:
      -redis-cluster_data-3:/bitnami/redis/data
    environment:
      -'REDIS_PASSWORD=123'
      -'REDIS_NODES=redis-node-0 redis-node-1 redis-node-2 redis-node-3 redis-node-4 redis-node-5'

  redis-node-4:
    image:docker.io/bitnami/redis-cluster:7.0
    container_name:node-4
    volumes:
      -redis-cluster_data-4:/bitnami/redis/data
    environment:
      -'REDIS_PASSWORD=123'
      -'REDIS_NODES=redis-node-0 redis-node-1 redis-node-2 redis-node-3 redis-node-4 redis-node-5'

  redis-node-5:
    image:docker.io/bitnami/redis-cluster:7.0
    container_name:node-5
    volumes:
      -redis-cluster_data-5:/bitnami/redis/data
    depends_on:
      -redis-node-0
      -redis-node-1
      -redis-node-2
      -redis-node-3
      -redis-node-4
    environment:
      -'REDIS_PASSWORD=123'
      -'REDISCLI_AUTH=123'
      -'REDIS_CLUSTER_REPLICAS=1'
      -'REDIS_NODES=redis-node-0 redis-node-1 redis-node-2 redis-node-3 redis-node-4 redis-node-5'
      -'REDIS_CLUSTER_CREATOR=yes'

volumes:
  redis-cluster_data-0:
    driver:local
  redis-cluster_data-1:
    driver:local
  redis-cluster_data-2:
    driver:local
  redis-cluster_data-3:
    driver:local
  redis-cluster_data-4:
    driver:local
  redis-cluster_data-5:
    driver:local

看到如下所示，即可認為redis cluster集群成功啟動。

?  ivansli$ docker-compose up -d
[+] Running 6/6
? Container node-2  Started                                                                                                                                        9.2s
? Container node-3  Started                                                                                                                                       10.6s
? Container node-1  Started                                                                                                                                        9.0s
? Container node-4  Started                                                                                                                                       11.4s
? Container node-0  Started                                                                                                                                        9.8s
? Container node-5  Started                                                                                                                                       21.5s
?  ivansli$ docker exec -it node-0 bash
ivansli@17affe689ed6:/$ redis-cli
127.0.0.1:6379> auth 123
OK
127.0.0.1:6379> cluster nodes
9ecc22a32815eb4f769069541279569c36b4b1d9 172.26.0.7:6379@16379 slave cc6e6b899e6f2b90f829aeff9ec36316433258a6 0 1685681651000 2 connected
f36960e21b778e1483237504fbffa2086ede7f4d 172.26.0.4:6379@16379 master - 0 1685681652000 3 connected 10923-16383
cc6e6b899e6f2b90f829aeff9ec36316433258a6 172.26.0.5:6379@16379 master - 0 1685681651976 2 connected 5461-10922
4523ca5ef7bfa6b7a22e39a418e2417204d5f34f 172.26.0.2:6379@16379 slave f36960e21b778e1483237504fbffa2086ede7f4d 0 1685681651000 3 connected
9cc6263d5ea15dcfb8795ce5e19f7844c5958781 172.26.0.3:6379@16379 slave 5bb46d5bb5df18a1977260da6183ec1cef9b8b00 0 1685681652984 1 connected
5bb46d5bb5df18a1977260da6183ec1cef9b8b00 172.26.0.6:6379@16379 myself,master - 0 1685681650000 1 connected 0-5460

可以看到redis cluster集群啟動成功，其架構為：3主(master)3從(slave)。

cluster master節點	slot范圍
master1	0-5460
master2	5461-10922
master3	10923-16383

集群共有16384(0~16383,2^14個)個slot(哈希槽)，每個master節點擁有連續的一段slot。

redis cluster集群key的存儲方式

眾所周知，對于單臺服務器來說，在計算機硬件不變更的情況下其資源空間存在上限，例如：內存、硬盤、網卡帶寬、CPU核心等。對于基于內存的redis服務來說，想要存儲更多的、超過單機內存容量的數據，那么只能采用集群模式，最常用的就是redis cluster集群。

其數據存儲原理簡化為公式：CRC16(key) % 16384 = slot，即：取key進行CRC16計算之后對16384取模運算得到key所在的slot，由于redis cluster在啟動時會對每一臺master節點分配slot空間，那么當前slot的值在哪臺master節點的slot空間范圍內，key就存儲在哪臺節點。

通俗的講：使用一種算法把整個數據空間化整為零分散存儲在多個節點 (分布式系統設計中常用策略)。

什么是hash tag

Redis官方對Hash tags的定義如下：

There is an exception for the computation of the hash slot that is used in order to implement hash tags. Hash tags are a way to ensure that multiple keys are allocated in the same hash slot. This is used in order to implement multi-key operations in Redis Cluster. To implement hash tags, the hash slot for a key is computed in a slightly different way in certain conditions. If the key contains a "{...}" pattern only the substring between { and } is hashed in order to obtain the hash slot. However since it is possible that there are multiple occurrences of { or } the algorithm is well specified by the following rules:

IF the key contains a { character.
AND IF there is a } character to the right of {.
AND IF there are one or more characters between the first occurrence of { and the first occurrence of }. Then instead of hashing the key, only what is between the first occurrence of { and the following first occurrence of } is hashed.

hash tags 是用于計算哈希槽時的一個例外，是一種確保多個鍵分配到同一個哈希槽中的方法。這是為了在Redis集群中實現多鍵操作而使用的。為了實現hash tags，在某些情況下，會以稍微不同的方式計算key的哈希槽。如果key包含"{...}"模式，則僅對{和}之間的子字符串進行散列以獲取哈希槽。但由于可能存在多個{或}出現，因此該算法遵循以下規則：

如果key包含字符 {
并且如果 } 字符位于 { 的右側
并且在第一個 { 和第一個 } 之間存在一個或多個字符

對于符合上述規則的key，則不會對整個key進行散列處理，而只會對第一次出現 { 和隨后第一次出現 } 之間的內容進行散列。否則，對整個key進行散列處理。

為什么使用hash tag

不使用hash tag批量獲取不同名稱的key

127.0.0.1:6379> mget name name1 name2 name3
(error) CROSSSLOT Keys in request don't hash to the same slot

顯示錯誤信息：CROSSSLOT 請求中的key沒有哈希到同一個插槽。

使用hash tag批量獲取不同名稱的key

172.26.0.5:6379> mget name {name} {name}1 {name}2 {name}3
1) (nil)
2) (nil)
3) (nil)
4) (nil)
5) (nil)

顯示正常(請求中的key被哈希到同一個插槽)。

對上述操作的key計算得出對應slot，整理如下所示：

key	計算key的slot	slot值	slot所在節點	是否使用hash tag
name	cluster keyslot name	5798	master2	?
name1	cluster keyslot name1	12933	master3	?
name2	cluster keyslot name2	742	master1	?
name3	cluster keyslot name3	4807	master1	?
{name}	cluster keyslot {name}	5798	master2	?
{name}1	cluster keyslot {name}1	5798	master2	?
{name}2	cluster keyslot {name}2	5798	master2	?
{name}3	cluster keyslot {name}3	5798	master2	?

redis中使用 cluster keyslot key的名稱可以得到key對應的slot值

結合上面例子以及官方對hash tag的描述，想必大家已經基本可以對為什么使用hash tag這個問題得出自己的結論。

假如在開發過程中，遇到Redis中既要存儲大量數據，又要讓某些相同特征的key(key包含相同字符串)存儲在同一個節點的情況。那么，這個時候Redis cluster+hash tag絕對是你的首選。

redis源碼中hash tag的計算過程

獲取key的hash tag值，主要包含2部分邏輯：

查找 {} 包含的字符串
對找到的字符串進行使用crc16()計算

具體代碼實現如在所示：

// 代碼文件：src/cluster.c

/* We have 16384 hash slots. The hash slot of a given key is obtained
 * as the least significant 14 bits of the crc16 of the key.
 *
 * However if the key contains the {...} pattern, only the part between
 * { and } is hashed. This may be useful in the future to force certain
 * keys to be in the same node (assuming no resharding is in progress). */
unsigned int keyHashSlot(char *key, int keylen) {
    int s, e; /* start-end indexes of { and } */

    // 找到第一個 { 出現的位置
    for (s = 0; s < keylen; s++)
        if (key[s] == '{') break;

    /* No '{' ? Hash the whole key. This is the base case. */
    /* 0x3FFF = 16383 = 11111111111111 (14個1)，使用 & 運算比取模運算 % 效率更高，并能實現相同效果*/
    // 沒有找到 { ，則使用整個key計算hash值
    if (s == keylen) return crc16(key,keylen) & 0x3FFF;

    // 找到第一個 } 出現的位置
    /* '{' found? Check if we have the corresponding '}'. */
    for (e = s+1; e < keylen; e++)
        if (key[e] == '}') break;

    /* No '}' or nothing between {} ? Hash the whole key. */
    // 沒找到 } 或者 {} 包含的字符串為空，則使用整個key計算hash值
    if (e == keylen || e == s+1) return crc16(key,keylen) & 0x3FFF;

    /* If we are here there is both a { and a } on its right. Hash
     * what is in the middle between { and }. */
    // 使用 {} 包含的字符串計算hash值
    return crc16(key+s+1,e-s-1) & 0x3FFF;
}


// 代碼文件：src/crc16.c
// crc16() 的實現邏輯

/* CRC16 implementation according to CCITT standards.
 *
 * Note by @antirez: this is actually the XMODEM CRC 16 algorithm, using the
 * following parameters:
 *
 * Name                       : "XMODEM", also known as "ZMODEM", "CRC-16/ACORN"
 * Width                      : 16 bit
 * Poly                       : 1021 (That is actually x^16 + x^12 + x^5 + 1)
 * Initialization             : 0000
 * Reflect Input byte         : False
 * Reflect Output CRC         : False
 * Xor constant to output CRC : 0000
 * Output for "123456789"     : 31C3
 */

staticconstuint16_t crc16tab[256]= {
    0x0000,0x1021,0x2042,0x3063,0x4084,0x50a5,0x60c6,0x70e7,
    0x8108,0x9129,0xa14a,0xb16b,0xc18c,0xd1ad,0xe1ce,0xf1ef,
    0x1231,0x0210,0x3273,0x2252,0x52b5,0x4294,0x72f7,0x62d6,
    0x9339,0x8318,0xb37b,0xa35a,0xd3bd,0xc39c,0xf3ff,0xe3de,
    0x2462,0x3443,0x0420,0x1401,0x64e6,0x74c7,0x44a4,0x5485,
    0xa56a,0xb54b,0x8528,0x9509,0xe5ee,0xf5cf,0xc5ac,0xd58d,
    0x3653,0x2672,0x1611,0x0630,0x76d7,0x66f6,0x5695,0x46b4,
    0xb75b,0xa77a,0x9719,0x8738,0xf7df,0xe7fe,0xd79d,0xc7bc,
    0x48c4,0x58e5,0x6886,0x78a7,0x0840,0x1861,0x2802,0x3823,
    0xc9cc,0xd9ed,0xe98e,0xf9af,0x8948,0x9969,0xa90a,0xb92b,
    0x5af5,0x4ad4,0x7ab7,0x6a96,0x1a71,0x0a50,0x3a33,0x2a12,
    0xdbfd,0xcbdc,0xfbbf,0xeb9e,0x9b79,0x8b58,0xbb3b,0xab1a,
    0x6ca6,0x7c87,0x4ce4,0x5cc5,0x2c22,0x3c03,0x0c60,0x1c41,
    0xedae,0xfd8f,0xcdec,0xddcd,0xad2a,0xbd0b,0x8d68,0x9d49,
    0x7e97,0x6eb6,0x5ed5,0x4ef4,0x3e13,0x2e32,0x1e51,0x0e70,
    0xff9f,0xefbe,0xdfdd,0xcffc,0xbf1b,0xaf3a,0x9f59,0x8f78,
    0x9188,0x81a9,0xb1ca,0xa1eb,0xd10c,0xc12d,0xf14e,0xe16f,
    0x1080,0x00a1,0x30c2,0x20e3,0x5004,0x4025,0x7046,0x6067,
    0x83b9,0x9398,0xa3fb,0xb3da,0xc33d,0xd31c,0xe37f,0xf35e,
    0x02b1,0x1290,0x22f3,0x32d2,0x4235,0x5214,0x6277,0x7256,
    0xb5ea,0xa5cb,0x95a8,0x8589,0xf56e,0xe54f,0xd52c,0xc50d,
    0x34e2,0x24c3,0x14a0,0x0481,0x7466,0x6447,0x5424,0x4405,
    0xa7db,0xb7fa,0x8799,0x97b8,0xe75f,0xf77e,0xc71d,0xd73c,
    0x26d3,0x36f2,0x0691,0x16b0,0x6657,0x7676,0x4615,0x5634,
    0xd94c,0xc96d,0xf90e,0xe92f,0x99c8,0x89e9,0xb98a,0xa9ab,
    0x5844,0x4865,0x7806,0x6827,0x18c0,0x08e1,0x3882,0x28a3,
    0xcb7d,0xdb5c,0xeb3f,0xfb1e,0x8bf9,0x9bd8,0xabbb,0xbb9a,
    0x4a75,0x5a54,0x6a37,0x7a16,0x0af1,0x1ad0,0x2ab3,0x3a92,
    0xfd2e,0xed0f,0xdd6c,0xcd4d,0xbdaa,0xad8b,0x9de8,0x8dc9,
    0x7c26,0x6c07,0x5c64,0x4c45,0x3ca2,0x2c83,0x1ce0,0x0cc1,
    0xef1f,0xff3e,0xcf5d,0xdf7c,0xaf9b,0xbfba,0x8fd9,0x9ff8,
    0x6e17,0x7e36,0x4e55,0x5e74,0x2e93,0x3eb2,0x0ed1,0x1ef0
};

uint16_t crc16(const char *buf, int len) {
    int counter;
    uint16_t crc = 0;
    for (counter = 0; counter < len; counter++)
            crc = (crc<<8) ^ crc16tab[((crc>>8) ^ *buf++)&0x00FF];
    return crc;
}

hash tag 可能導致的問題

在Redis cluster集群中，使用hash tag可以確保多個鍵分配到同一個哈希槽中。但是，以下幾種情況可能會導致一些負面影響：

存儲時，大量key使用hash tag后落到同一slot，slot所在節點導致存儲大量數據，甚至超過內存上限（數據傾斜）
查詢/刪除時，大量key使用hash tag后落到同一slot，slot所在節點處理大量請求，導致服務器忙碌、響應失敗，甚至宕機（hot key）

責任編輯：武曉燕來源：編程技術之道

Redis 場景內存