记录一次阿里云flink生产故障_error during binlog processing. last offset stored

作者：码创造者 | 2024-07-16 21:43:04
踩
error during binlog processing. last offset stored = null, binlog reader nea
2022.12.9下午三点半发现flink监控报警一直显示作业反复重启,打开管理平台发现作业在不断自动重启,找到了第一行报错日志.
报错日志如下

2022-12-09 15:29:28,263 ERROR io.debezium.connector.mysql.MySqlStreamingChangeEventSource  [] - Error during binlog processing. Last offset stored = null, binlog reader near position = mysql-bin.001675/185277142
2022-12-09 15:29:28,264 ERROR io.debezium.pipeline.ErrorHandler                            [] - Producer failure
io.debezium.DebeziumException: null
	at io.debezium.connector.mysql.MySqlStreamingChangeEventSource.wrap(MySqlStreamingChangeEventSource.java:1146) ~[blob_p-ab098b2976a0009b1eaabcfae48660640b2f63ab-efc656c3c9ffddd2404de700a0f810c2:?]
	at io.debezium.connector.mysql.MySqlStreamingChangeEventSource$ReaderThreadLifecycleListener.onEventDeserializationFailure(MySqlStreamingChangeEventSource.java:1193) [blob_p-ab098b2976a0009b1eaabcfae48660640b2f63ab-efc656c3c9ffddd2404de700a0f810c2:?]
	at com.github.shyiko.mysql.binlog.BinaryLogClient.listenForEventPackets(BinaryLogClient.java:958) [blob_p-ab098b2976a0009b1eaabcfae48660640b2f63ab-efc656c3c9ffddd2404de700a0f810c2:?]
	at com.github.shyiko.mysql.binlog.BinaryLogClient.connect(BinaryLogClient.java:606) [blob_p-ab098b2976a0009b1eaabcfae48660640b2f63ab-efc656c3c9ffddd2404de700a0f810c2:?]
	at com.github.shyiko.mysql.binlog.BinaryLogClient$7.run(BinaryLogClient.java:850) [blob_p-ab098b2976a0009b1eaabcfae48660640b2f63ab-efc656c3c9ffddd2404de700a0f810c2:?]
	at java.lang.Thread.run(Thread.java:877) [?:1.8.0_302]
Caused by: java.lang.NullPointerException
	at com.github.shyiko.mysql.binlog.event.deserialization.TableMapEventDataDeserializer.readMetadata(TableMapEventDataDeserializer.java:80) ~[blob_p-ab098b2976a0009b1eaabcfae48660640b2f63ab-efc656c3c9ffddd2404de700a0f810c2:?]
	at com.github.shyiko.mysql.binlog.event.deserialization.TableMapEventDataDeserializer.deserialize(TableMapEventDataDeserializer.java:42) ~[blob_p-ab098b2976a0009b1eaabcfae48660640b2f63ab-efc656c3c9ffddd2404de700a0f810c2:?]
	at com.github.shyiko.mysql.binlog.event.deserialization.TableMapEventDataDeserializer.deserialize(TableMapEventDataDeserializer.java:27) ~[blob_p-ab098b2976a0009b1eaabcfae48660640b2f63ab-efc656c3c9ffddd2404de700a0f810c2:?]
	at com.github.shyiko.mysql.binlog.event.deserialization.EventDeserializer.deserializeEventData(EventDeserializer.java:303) ~[blob_p-ab098b2976a0009b1eaabcfae48660640b2f63ab-efc656c3c9ffddd2404de700a0f810c2:?]
	at com.github.shyiko.mysql.binlog.event.deserialization.EventDeserializer.deserializeTableMapEventData(EventDeserializer.java:281) ~[blob_p-ab098b2976a0009b1eaabcfae48660640b2f63ab-efc656c3c9ffddd2404de700a0f810c2:?]
	at com.github.shyiko.mysql.binlog.event.deserialization.EventDeserializer.nextEvent(EventDeserializer.java:228) ~[blob_p-ab098b2976a0009b1eaabcfae48660640b2f63ab-efc656c3c9ffddd2404de700a0f810c2:?]
	at io.debezium.connector.mysql.MySqlStreamingChangeEventSource$1.nextEvent(MySqlStreamingChangeEventSource.java:233) ~[blob_p-ab098b2976a0009b1eaabcfae48660640b2f63ab-efc656c3c9ffddd2404de700a0f810c2:?]
	at com.github.shyiko.mysql.binlog.BinaryLogClient.listenForEventPackets(BinaryLogClient.java:945) ~[blob_p-ab098b2976a0009b1eaabcfae48660640b2f63ab-efc656c3c9ffddd2404de700a0f810c2:?]
	... 3 more
2022-12-09 15:29:28,276 ERROR io.debezium.connector.mysql.MySqlStreamingChangeEventSource  [] - Error during binlog processing. Last offset stored = null, binlog reader near position = mysql-bin.001675/185277142
2022-12-09 15:29:28,276 ERROR io.debezium.pipeline.ErrorHandler                            [] - Producer failure
io.debezium.DebeziumException: Failed to deserialize data of EventHeaderV4{timestamp=1670570968000, eventType=UPDATE_ROWS, serverId=1938206218, headerLength=19, dataLength=734, nextPosition=185278024, flags=0}
	at io.debezium.connector.mysql.MySqlStreamingChangeEventSource.wrap(MySqlStreamingChangeEventSource.java:1146) ~[blob_p-ab098b2976a0009b1eaabcfae48660640b2f63ab-efc656c3c9ffddd2404de700a0f810c2:?]
	at io.debezium.connector.mysql.MySqlStreamingChangeEventSource$ReaderThreadLifecycleListener.onEventDeserializationFailure(MySqlStreamingChangeEventSource.java:1193) [blob_p-ab098b2976a0009b1eaabcfae48660640b2f63ab-efc656c3c9ffddd2404de700a0f810c2:?]
	at com.github.shyiko.mysql.binlog.BinaryLogClient.listenForEventPackets(BinaryLogClient.java:958) [blob_p-ab098b2976a0009b1eaabcfae48660640b2f63ab-efc656c3c9ffddd2404de700a0f810c2:?]
	at com.github.shyiko.mysql.binlog.BinaryLogClient.connect(BinaryLogClient.java:606) [blob_p-ab098b2976a0009b1eaabcfae48660640b2f63ab-efc656c3c9ffddd2404de700a0f810c2:?]
	at com.github.shyiko.mysql.binlog.BinaryLogClient$7.run(BinaryLogClient.java:850) [blob_p-ab098b2976a0009b1eaabcfae48660640b2f63ab-efc656c3c9ffddd2404de700a0f810c2:?]
	at java.lang.Thread.run(Thread.java:877) [?:1.8.0_302]
Caused by: com.github.shyiko.mysql.binlog.event.deserialization.EventDataDeserializationException: Failed to deserialize data of EventHeaderV4{timestamp=1670570968000, eventType=UPDATE_ROWS, serverId=1938206218, headerLength=19, dataLength=734, nextPosition=185278024, flags=0}
	at com.github.shyiko.mysql.binlog.event.deserialization.EventDeserializer.deserializeEventData(EventDeserializer.java:309) ~[blob_p-ab098b2976a0009b1eaabcfae48660640b2f63ab-efc656c3c9ffddd2404de700a0f810c2:?]
	at com.github.shyiko.mysql.binlog.event.deserialization.EventDeserializer.nextEvent(EventDeserializer.java:232) ~[blob_p-ab098b2976a0009b1eaabcfae48660640b2f63ab-efc656c3c9ffddd2404de700a0f810c2:?]
	at io.debezium.connector.mysql.MySqlStreamingChangeEventSource$1.nextEvent(MySqlStreamingChangeEventSource.java:233) ~[blob_p-ab098b2976a0009b1eaabcfae48660640b2f63ab-efc656c3c9ffddd2404de700a0f810c2:?]
	at com.github.shyiko.mysql.binlog.BinaryLogClient.listenForEventPackets(BinaryLogClient.java:945) ~[blob_p-ab098b2976a0009b1eaabcfae48660640b2f63ab-efc656c3c9ffddd2404de700a0f810c2:?]
	... 3 more
Caused by: com.github.shyiko.mysql.binlog.event.deserialization.MissingTableMapEventException: No TableMapEventData has been found for table id:406820. Usually that means that you have started reading binary log 'within the logical event group' (e.g. from WRITE_ROWS and not proceeding TABLE_MAP
	at com.github.shyiko.mysql.binlog.event.deserialization.AbstractRowsEventDataDeserializer.deserializeRow(AbstractRowsEventDataDeserializer.java:109) ~[blob_p-ab098b2976a0009b1eaabcfae48660640b2f63ab-efc656c3c9ffddd2404de700a0f810c2:?]
	at com.github.shyiko.mysql.binlog.event.deserialization.UpdateRowsEventDataDeserializer.deserializeRows(UpdateRowsEventDataDeserializer.java:71) ~[blob_p-ab098b2976a0009b1eaabcfae48660640b2f63ab-efc656c3c9ffddd2404de700a0f810c2:?]
	at com.github.shyiko.mysql.binlog.event.deserialization.UpdateRowsEventDataDeserializer.deserialize(UpdateRowsEventDataDeserializer.java:58) ~[blob_p-ab098b2976a0009b1eaabcfae48660640b2f63ab-efc656c3c9ffddd2404de700a0f810c2:?]
	at com.github.shyiko.mysql.binlog.event.deserialization.UpdateRowsEventDataDeserializer.deserialize(UpdateRowsEventDataDeserializer.java:33) ~[blob_p-ab098b2976a0009b1eaabcfae48660640b2f63ab-efc656c3c9ffddd2404de700a0f810c2:?]
	at com.github.shyiko.mysql.binlog.event.deserialization.EventDeserializer.deserializeEventData(EventDeserializer.java:303) ~[blob_p-ab098b2976a0009b1eaabcfae48660640b2f63ab-efc656c3c9ffddd2404de700a0f810c2:?]
	at com.github.shyiko.mysql.binlog.event.deserialization.EventDeserializer.nextEvent(EventDeserializer.java:232) ~[blob_p-ab098b2976a0009b1eaabcfae48660640b2f63ab-efc656c3c9ffddd2404de700a0f810c2:?]
	at io.debezium.connector.mysql.MySqlStreamingChangeEventSource$1.nextEvent(MySqlStreamingChangeEventSource.java:233) ~[blob_p-ab098b2976a0009b1eaabcfae48660640b2f63ab-efc656c3c9ffddd2404de700a0f810c2:?]
	at com.github.shyiko.mysql.binlog.BinaryLogClient.listenForEventPackets(BinaryLogClient.java:945) ~[blob_p-ab098b2976a0009b1eaabcfae48660640b2f63ab-efc656c3c9ffddd2404de700a0f810c2:?]
	... 3 more
2022-12-09 15:29:28,665 ERROR org.apache.flink.connector.base.source.reader.fetcher.SplitFetcherManager [] - Received uncaught exception.
java.lang.RuntimeException: SplitFetcher thread 768 received unexpected exception while polling the records
	at org.apache.flink.connector.base.source.reader.fetcher.SplitFetcher.runOnce(SplitFetcher.java:146) ~[flink-table_2.11-1.13-vvr-4.0.14-2-SNAPSHOT.jar:1.13-vvr-4.0.14-2-SNAPSHOT]
	at org.apache.flink.connector.base.source.reader.fetcher.SplitFetcher.run(SplitFetcher.java:101) [flink-table_2.11-1.13-vvr-4.0.14-2-SNAPSHOT.jar:1.13-vvr-4.0.14-2-SNAPSHOT]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_302]
	at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_302]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_302]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_302]
	at java.lang.Thread.run(Thread.java:877) [?:1.8.0_302]
Caused by: org.apache.kafka.connect.errors.ConnectException: An exception occurred in the change event producer. This connector will be stopped.
	at io.debezium.pipeline.ErrorHandler.setProducerThrowable(ErrorHandler.java:42) ~[blob_p-ab098b2976a0009b1eaabcfae48660640b2f63ab-efc656c3c9ffddd2404de700a0f810c2:?]
	at com.ververica.cdc.connectors.mysql.debezium.task.context.MySqlErrorHandler.setProducerThrowable(MySqlErrorHandler.java:72) ~[blob_p-ab098b2976a0009b1eaabcfae48660640b2f63ab-efc656c3c9ffddd2404de700a0f810c2:?]
	at io.debezium.connector.mysql.MySqlStreamingChangeEventSource$ReaderThreadLifecycleListener.onEventDeserializationFailure(MySqlStreamingChangeEventSource.java:1193) ~[blob_p-ab098b2976a0009b1eaabcfae48660640b2f63ab-efc656c3c9ffddd2404de700a0f810c2:?]
	at com.github.shyiko.mysql.binlog.BinaryLogClient.listenForEventPackets(BinaryLogClient.java:958) ~[blob_p-ab098b2976a0009b1eaabcfae48660640b2f63ab-efc656c3c9ffddd2404de700a0f810c2:?]
	at com.github.shyiko.mysql.binlog.BinaryLogClient.connect(BinaryLogClient.java:606) ~[blob_p-ab098b2976a0009b1eaabcfae48660640b2f63ab-efc656c3c9ffddd2404de700a0f810c2:?]
	at com.github.shyiko.mysql.binlog.BinaryLogClient$7.run(BinaryLogClient.java:850) ~[blob_p-ab098b2976a0009b1eaabcfae48660640b2f63ab-efc656c3c9ffddd2404de700a0f810c2:?]
	... 1 more
Caused by: io.debezium.DebeziumException
	at io.debezium.connector.mysql.MySqlStreamingChangeEventSource.wrap(MySqlStreamingChangeEventSource.java:1146) ~[blob_p-ab098b2976a0009b1eaabcfae48660640b2f63ab-efc656c3c9ffddd2404de700a0f810c2:?]
	at io.debezium.connector.mysql.MySqlStreamingChangeEventSource$ReaderThreadLifecycleListener.onEventDeserializationFailure(MySqlStreamingChangeEventSource.java:1193) ~[blob_p-ab098b2976a0009b1eaabcfae48660640b2f63ab-efc656c3c9ffddd2404de700a0f810c2:?]
	at com.github.shyiko.mysql.binlog.BinaryLogClient.listenForEventPackets(BinaryLogClient.java:958) ~[blob_p-ab098b2976a0009b1eaabcfae48660640b2f63ab-efc656c3c9ffddd2404de700a0f810c2:?]
	at com.github.shyiko.mysql.binlog.BinaryLogClient.connect(BinaryLogClient.java:606) ~[blob_p-ab098b2976a0009b1eaabcfae48660640b2f63ab-efc656c3c9ffddd2404de700a0f810c2:?]
	at com.github.shyiko.mysql.binlog.BinaryLogClient$7.run(BinaryLogClient.java:850) ~[blob_p-ab098b2976a0009b1eaabcfae48660640b2f63ab-efc656c3c9ffddd2404de700a0f810c2:?]
	... 1 more
Caused by: java.lang.NullPointerException
	at com.github.shyiko.mysql.binlog.event.deserialization.TableMapEventDataDeserializer.readMetadata(TableMapEventDataDeserializer.java:80) ~[blob_p-ab098b2976a0009b1eaabcfae48660640b2f63ab-efc656c3c9ffddd2404de700a0f810c2:?]
	at com.github.shyiko.mysql.binlog.event.deserialization.TableMapEventDataDeserializer.deserialize(TableMapEventDataDeserializer.java:42) ~[blob_p-ab098b2976a0009b1eaabcfae48660640b2f63ab-efc656c3c9ffddd2404de700a0f810c2:?]
	at com.github.shyiko.mysql.binlog.event.deserialization.TableMapEventDataDeserializer.deserialize(TableMapEventDataDeserializer.java:27) ~[blob_p-ab098b2976a0009b1eaabcfae48660640b2f63ab-efc656c3c9ffddd2404de700a0f810c2:?]
	at com.github.shyiko.mysql.binlog.event.deserialization.EventDeserializer.deserializeEventData(EventDeserializer.java:303) ~[blob_p-ab098b2976a0009b1eaabcfae48660640b2f63ab-efc656c3c9ffddd2404de700a0f810c2:?]
	at com.github.shyiko.mysql.binlog.event.deserialization.EventDeserializer.deserializeTableMapEventData(EventDeserializer.java:281) ~[blob_p-ab098b2976a0009b1eaabcfae48660640b2f63ab-efc656c3c9ffddd2404de700a0f810c2:?]
	at com.github.shyiko.mysql.binlog.event.deserialization.EventDeserializer.nextEvent(EventDeserializer.java:228) ~[blob_p-ab098b2976a0009b1eaabcfae48660640b2f63ab-efc656c3c9ffddd2404de700a0f810c2:?]
	at io.debezium.connector.mysql.MySqlStreamingChangeEventSource$1.nextEvent(MySqlStreamingChangeEventSource.java:233) ~[blob_p-ab098b2976a0009b1eaabcfae48660640b2f63ab-efc656c3c9ffddd2404de700a0f810c2:?]
	at com.github.shyiko.mysql.binlog.BinaryLogClient.listenForEventPackets(BinaryLogClient.java:945) ~[blob_p-ab098b2976a0009b1eaabcfae48660640b2f63ab-efc656c3c9ffddd2404de700a0f810c2:?]
	at com.github.shyiko.mysql.binlog.BinaryLogClient.connect(BinaryLogClient.java:606) ~[blob_p-ab098b2976a0009b1eaabcfae48660640b2f63ab-efc656c3c9ffddd2404de700a0f810c2:?]
	at com.github.shyiko.mysql.binlog.BinaryLogClient$7.run(BinaryLogClient.java:850) ~[blob_p-ab098b2976a0009b1eaabcfae48660640b2f63ab-efc656c3c9ffddd2404de700a0f810c2:?]
	... 1 more
TableMapEventDataDeserializer,第80行代码如下
ColumnType枚举如下
拉取到mysql-bin.001675/185277142,用阿里云开发提供的测试类读取,打断点进入发现columnTypes[i] & 0xFF的值为20,并不在上述枚举类中,switch(null)会导致报错.
在github上找到了相关issue,原因是mysql8.0.18多了一个类型MYSQL_TYPE_TYPED_ARRAY，这个是不可见的，然后他的ID从244变成了20,导致解析不了,检查了生产的mysql配置,发现确实是8.0.18.
https://github.com/mysql/mysql-server/commit/9082b6a820f3948fd563cc32a050f5e8775f2855
接下来就是等待阿里云flink发布新版vvr解决问题了
声明：本文内容由网友自发贡献，不代表【wpsshop博客】立场，版权归原作者所有，本站不承担相应法律责任。如您发现有侵权的内容，请联系我们。转载请注明出处：https://www.wpsshop.cn/w/码创造者/article/detail/836303