Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

utf8 not supported #2

Open
hamtie opened this issue Mar 23, 2012 · 2 comments · May be fixed by #3
Open

utf8 not supported #2

hamtie opened this issue Mar 23, 2012 · 2 comments · May be fixed by #3

Comments

@hamtie
Copy link

hamtie commented Mar 23, 2012

I just noticed that utf8 isn't supported. I can't tell if this is occurring in the crc32 hashing algorithm or in the packing library. Here's what I've used to verify this behavior:

Producer initialization

var kafka = require('kafka');
var kclient = new kafka.Producer();

Message with utf char that doesn't send

kclient.send('test-raw',  'Guantánamo');

Remove the char and it sends:

kclient.send('test-raw',   'Guantnamo' );
@hamtie
Copy link
Author

hamtie commented Mar 23, 2012

I should clarify that the message with utf chars does actually make it to the kafka server but fails with an InvalidMessageException

[2012-03-22 20:10:23,078] ERROR Error processing ProduceRequest on test-raw:0 (kafka.server.KafkaRequestHandlers)
kafka.message.InvalidMessageException
    at kafka.log.Log$$anonfun$append$1.apply(Log.scala:204)
    at kafka.log.Log$$anonfun$append$1.apply(Log.scala:202)
    at scala.collection.Iterator$class.foreach(Iterator.scala:631)
    at kafka.utils.IteratorTemplate.foreach(IteratorTemplate.scala:30)
    at scala.collection.IterableLike$class.foreach(IterableLike.scala:79)
    at kafka.message.MessageSet.foreach(MessageSet.scala:87)
    at kafka.log.Log.append(Log.scala:202)
    at kafka.server.KafkaRequestHandlers.kafka$server$KafkaRequestHandlers$$handleProducerRequest(KafkaRequestHandlers.scala:75)
    at kafka.server.KafkaRequestHandlers.handleProducerRequest(KafkaRequestHandlers.scala:58)
    at kafka.server.KafkaRequestHandlers$$anonfun$handlerFor$1.apply(KafkaRequestHandlers.scala:43)
    at kafka.server.KafkaRequestHandlers$$anonfun$handlerFor$1.apply(KafkaRequestHandlers.scala:43)
    at kafka.network.Processor.handle(SocketServer.scala:289)
    at kafka.network.Processor.read(SocketServer.scala:312)
    at kafka.network.Processor.run(SocketServer.scala:207)
    at java.lang.Thread.run(Thread.java:662)
[2012-03-22 20:10:23,078] ERROR Closing socket for /127.0.0.1 because of error (kafka.network.Processor)
kafka.message.InvalidMessageException
    at kafka.log.Log$$anonfun$append$1.apply(Log.scala:204)
    at kafka.log.Log$$anonfun$append$1.apply(Log.scala:202)
    at scala.collection.Iterator$class.foreach(Iterator.scala:631)
    at kafka.utils.IteratorTemplate.foreach(IteratorTemplate.scala:30)
    at scala.collection.IterableLike$class.foreach(IterableLike.scala:79)
    at kafka.message.MessageSet.foreach(MessageSet.scala:87)
    at kafka.log.Log.append(Log.scala:202)
    at kafka.server.KafkaRequestHandlers.kafka$server$KafkaRequestHandlers$$handleProducerRequest(KafkaRequestHandlers.scala:75)
    at kafka.server.KafkaRequestHandlers.handleProducerRequest(KafkaRequestHandlers.scala:58)
    at kafka.server.KafkaRequestHandlers$$anonfun$handlerFor$1.apply(KafkaRequestHandlers.scala:43)
    at kafka.server.KafkaRequestHandlers$$anonfun$handlerFor$1.apply(KafkaRequestHandlers.scala:43)
    at kafka.network.Processor.handle(SocketServer.scala:289)
    at kafka.network.Processor.read(SocketServer.scala:312)
    at kafka.network.Processor.run(SocketServer.scala:207)
    at java.lang.Thread.run(Thread.java:662)

In the scala source code for InvalidMessageException, it says that the exception

Indicates that a message failed its checksum and is corrupt

@hamtie
Copy link
Author

hamtie commented Mar 23, 2012

I've investigated the problem a little more and it looks like the problem must be in std.pack or possibly a fault in how the message is created. Creating the crc32 output from a core library in python I get

import zlib
zlib.crc32('Guantnamo')
882229988

zlib.crc32('Guantánamo')
1464054322

which is the same in std.crc32

var std = require("std")
std.crc32('Guantnamo')
882229988

std.crc32('Guantánamo')
1464054322

@hamtie hamtie linked a pull request Mar 23, 2012 that will close this issue
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Send exactly the same message to kafka that's used in the hash function.... hamtie/node-kafka
1 participant
@hamtie