Around 30 billion messages are sent using WhatsApp by its 990 million users daily. Among them 70% are adult. Since it’s launch in 2009, WhatsApp has takeover the messaging market. But, who is ruling this gigantic empire?. What is the technology stack behind the scene ??, How does it works??? ……..
In this part of “Inside of WhatsApp” series, I will introduce you with technology stack whatsApp is using to provide it’s non breakable service, A customized XMPP protocol to make communication faster, and WhatsApp registration flow.
1) Technology Stack
* Operating System: FreeBSD
* Server: Yaws
* Server Application: custom ejaberd
* Language : erlang
* Technology : custom XMPP (funXMPP)
* Web: PHP ( WhatsAppWeb)
* Database : mnesia
* Encryption : RC4
2) FunXMPP
WhatsApp uses a protocol which is a slimmed-down version of XMPP. I won’t explain the entire XMPP protocol (see RFC 6120, 6121, 3920, 3921, 3922, and 3923 for that), but it is a messaging protocol using XML as its syntax. A simple example of an XMPP message would be:
<message to="34123456789@s.whatsapp.net" type="text" id="message-1417651059-2" t="1417651059">
<body>Test</body>
</message>
But apparently the creators of WhatsApp thought this was too bloated and found a way to express XMPP messages using only a few bytes, which they called FunXMPP. Since WhatsApp is intended for mobile devices which often lack a good internet connection, it is logical they wanted as few overhead as possible. Using FunXMPP they achieved that, while still using a standard internet protocol.
So how does FunXMPP accomplish this?
First of all, all keywords are assigned a byte. In the above example there are a lot of keywords which are common in xmpp (eg message, from, type, text).
If you can replace those by just one byte, it would reduce a lot of overhead. FunXMPP uses a HashTable for this, containing most (if not all) keywords.
Given the syntax \xnn for one byte with the hexadecimal value nn, the above example could be reduced to:
<\x59 \xa5="01234567890@\x91" \xa7="\xa2" \x44="message-1417651059-2" \xa1="1417651059">
<\x12>Test</\x12>
</\x58>
Keeping in mind that \xnn stands for just one byte, this is already a significant reduce in size. Note that all remaining ascii values (eg 1417651059, Test, message-1417651059-2) cannot be replaced by anything because they are variable (ie can be set by the user).
XML is a human readable format employing tags that must be opened and closed. Is this really necessary for a computer to read an XML structure? The creators of FunXMPP must have thought the same thing because the other method of decreasing the size of messages is the encoding of the XML structure as a few bytes.
The only thing that remains now is the XML structure. In FunXMPP this structure is expressed as a set of lists. A list is designated by a \xf8 byte. After this \xf8 byte comes a byte with the number of items the list contains. Things that count as one item here are: the tag name, keys, values and the body.
In general: a list followed directly by a list means there are several nodes at the same level and the first list is not a tag or anything visible in the XML.
- Token MapLookup
0x03 => 'account', 0x04 => 'ack', 0x05 => 'action', 0x06 => 'active', 0x07 => 'add', 0x08 => 'after', 0x09 => 'all', 0x0a => 'allow', 0x0b => 'apple', 0x0c => 'auth', 0x0d => 'author', 0x0e => 'available', 0x0f => 'bad-protocol', 0x10 => 'bad-request', 0x11 => 'before', 0x12 => 'body', 0x13 => 'broadcast', 0x14 => 'cancel', 0x15 => 'category', 0x16 => 'challenge', 0x17 => 'chat', 0x18 => 'clean', 0x19 => 'code', 0x1a => 'composing', 0x1b => 'config', 0x1c => 'contacts', 0x1d => 'count', 0x1e => 'create', 0x1f => 'creation', 0x20 => 'debug', 0x21 => 'default', 0x22 => 'delete', 0x23 => 'delivery', 0x24 => 'delta', 0x25 => 'deny', 0x26 => 'digest', 0x27 => 'dirty', 0x28 => 'duplicate', 0x29 => 'elapsed', 0x2a => 'enable', 0x2b => 'encoding', 0x2c => 'error', 0x2d => 'event', 0x2e => 'expiration', 0x2f => 'expired', 0x30 => 'fail', 0x31 => 'failure', 0x32 => 'false', 0x33 => 'favorites', 0x34 => 'feature', 0x35 => 'features', 0x36 => 'feature-not-implemented', 0x37 => 'field', 0x38 => 'first', 0x39 => 'free', 0x3a => 'from', 0x3b => 'g.us', 0x3c => 'get', 0x3d => 'google', 0x3e => 'group', 0x3f => 'groups', 0x40 => 'groups_v2', 0x41 => 'http://etherx.jabber.org/streams', 0x42 => 'http://jabber.org/protocol/chatstates', 0x43 => 'ib', 0x44 => 'id', 0x45 => 'image', 0x46 => 'img', 0x47 => 'index', 0x48 => 'internal-server-error', 0x49 => 'ip', 0x4a => 'iq', 0x4b => 'item-not-found', 0x4c => 'item', 0x4d => 'jabber:iq:last', 0x4e => 'jabber:iq:privacy', 0x4f => 'jabber:x:event', 0x50 => 'jid', 0x51 => 'kind', 0x52 => 'last', 0x53 => 'leave', 0x54 => 'list', 0x55 => 'max', 0x56 => 'mechanism', 0x57 => 'media', 0x58 => 'message_acks', 0x59 => 'message', 0x5a => 'method', 0x5b => 'microsoft', 0x5c => 'missing', 0x5d => 'modify', 0x5e => 'mute', 0x5f => 'name', 0x60 => 'nokia', 0x61 => 'none', 0x62 => 'not-acceptable', 0x63 => 'not-allowed', 0x64 => 'not-authorized', 0x65 => 'notification', 0x66 => 'notify', 0x67 => 'off', 0x68 => 'offline', 0x69 => 'order', 0x6a => 'owner', 0x6b => 'owning', 0x6c => 'p_o', 0x6d => 'p_t', 0x6e => 'paid', 0x6f => 'participant', 0x70 => 'participants', 0x71 => 'participating', 0x72 => 'paused', 0x73 => 'picture', 0x74 => 'pin', 0x75 => 'ping', 0x76 => 'platform', 0x77 => 'port', 0x78 => 'presence', 0x79 => 'preview', 0x7a => 'probe', 0x7b => 'prop', 0x7c => 'props', 0x7d => 'query', 0x7e => 'raw', 0x7f => 'read', 0x80 => 'readreceipts', 0x81 => 'reason', 0x82 => 'receipt', 0x83 => 'relay', 0x84 => 'remote-server-timeout', 0x85 => 'remove', 0x86 => 'request', 0x87 => 'required', 0x88 => 'resource-constraint', 0x89 => 'resource', 0x8a => 'response', 0x8b => 'result', 0x8c => 'retry', 0x8d => 'rim', 0x8e => 's_o', 0x8f => 's_t', 0x90 => 's.us', 0x91 => 's.whatsapp.net', 0x92 => 'seconds', 0x93 => 'server-error', 0x94 => 'server', 0x95 => 'service-unavailable', 0x96 => 'set', 0x97 => 'show', 0x98 => 'silent', 0x99 => 'stat', 0x9a => 'status', 0x9b => 'stream:error', 0x9c => 'stream:features', 0x9d => 'subject', 0x9e => 'subscribe', 0x9f => 'success', 0xa0 => 'sync', 0xa1 => 't', 0xa2 => 'text', 0xa3 => 'timeout', 0xa4 => 'timestamp', 0xa5 => 'to', 0xa6 => 'true', 0xa7 => 'type', 0xa8 => 'unavailable', 0xa9 => 'unsubscribe', 0xaa => 'uri', 0xab => 'url', 0xac => 'urn:ietf:params:xml:ns:xmpp-sasl', 0xad => 'urn:ietf:params:xml:ns:xmpp-stanzas', 0xae => 'urn:ietf:params:xml:ns:xmpp-streams', 0xaf => 'urn:xmpp:ping', 0xb0 => 'urn:xmpp:whatsapp:account', 0xb1 => 'urn:xmpp:whatsapp:dirty', 0xb2 => 'urn:xmpp:whatsapp:mms', 0xb3 => 'urn:xmpp:whatsapp:push', 0xb4 => 'urn:xmpp:whatsapp', 0xb5 => 'user', 0xb6 => 'user-not-found', 0xb7 => 'value', 0xb8 => 'version', 0xb9 => 'w:g', 0xba => 'w:p:r', 0xbb => 'w:p', 0xbc => 'w:profile:picture', 0xbd => 'w', 0xbe => 'wait', 0xbf => 'WAUTH-2', 0xc0 => 'xmlns:stream', 0xc1 => 'xmlns', 0xc2 => '1', 0xc3 => 'chatstate', 0xc4 => 'crypto', 0xc5 => 'phash', 0xc6 => 'enc', 0xc7 => 'class', 0xc8 => 'off_cnt', 0xc9 => 'w:g2', 0xca => 'promote', 0xcb => 'demote', 0xcc => 'creator', 0xcd => 'Bell.caf', 0xce => 'Boing.caf', 0xcf => 'Glass.caf', 0xd0 => 'Harp.caf', 0xd1 => 'TimePassing.caf', 0xd2 => 'Tri-tone.caf', 0xd3 => 'Xylophone.caf', 0xd4 => 'background', 0xd5 => 'backoff', 0xd6 => 'chunked', 0xd7 => 'context', 0xd8 => 'full', 0xd9 => 'in', 0xda => 'interactive', 0xdb => 'out', 0xdc => 'registration', 0xdd => 'sid', 0xde => 'urn:xmpp:whatsapp:sync', 0xdf => 'flt', 0xe0 => 's16', 0xe1 => 'u8', 0xe2 => 'adpcm', 0xe3 => 'amrnb', 0xe4 => 'amrwb', 0xe5 => 'mp3', 0xe6 => 'pcm', 0xe7 => 'qcelp', 0xe8 => 'wma', 0xe9 => 'h263', 0xea => 'h264', 0xeb => 'jpeg'.
3) WhatsApp Registration Flow
WhatsApp registration process is very confusing or no official document is available for guidance . Request tracking using traffic monitoring tools has revealed following truth about registration process. Below is the simplified version of process flow.
- HTTP request made by client:Here is the inside look of HTTP request made by whats app client to server. Whats app operates it’s all services on “https://v.whatsapp.net” followed by version.
Requesting code
https://v.whatsapp.net/v2/code?in=*********&cc=**&id=%**%DC%E7%**%C*%A8%**%26%**%4F%35%**%B1%69%AF%**&lg=es&lc=ES&sim_mcc=214&sim_mnc=007&method=sms&token=*********TOKEN*********
- method: sms or voice
- in: phone number without country code
- cc: country code
- id: identity
- lg: Language (identified by checking sim_mnc and sim_mcc)
- lc: Language set in the device, if not found it will set
zz - token: token (used for requesting sms/voice code)
- sim_mcc: mcc of sim
- sim_mnc: mnc of sim
Registering number
https://v.whatsapp.net/v2/register?cc=**&in=*********&id=%**%DC%E7%**%C*%A8%**%26%**%4F%35%**%B1%69%AF%**&code=202357&lg=es&lc=ES
- cc: country code
- in: phone number without country code
- id: identity
- code: code received by sms or voice call
Checking if the number exists (Retrieve password if number was register in that device).
https://v.whatsapp.net/v2/exist?cc=**&in=*********&id=%**%DC%E7%**%C*%A8%**%26%**%4F%35%**%B1%69%AF%**&lg=es&lc=ES
- cc: country code
- in: phone number without country code
- id: identity
- lg: Language (identified by checking sim_mnc and sim_mcc)
- lc: Language set in the device, if not found it will set
zz
Sunny Patel 