LevelDB partitons keeps corrupting, riak node not running
LevelDB partitons keeps corrupting, riak node not running
I'm using Riak KV which is running in docker container. From time to time some LevelDB partitions has truncuated files and node keeps crashing. I can't run partition repair, so I've tried to delete corrupted partition and it works. I want to repair this problem, and get rid of corrupting partitions.
Here is part of crash log:
2018-07-30 15:24:22 =CRASH REPORT====
crasher:
initial call: riak_core_vnode:init/1
pid: <0.1290.0>
registered_name:
exception exit: {{function_clause,[{riak_kv_vnode,terminate,[{bad_return_value,{stop,{db_open,"Corruption: truncated record at end of file"}}},undefined],[{file,"src/riak_kv_vnode.erl"},{line,1066}]},{riak_core_vnode,terminate,3,[{file,"src/riak_core_vnode.erl"},{line,907}]},{gen_fsm,terminate,7,[{file,"gen_fsm.erl"},{line,597}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,239}]}]},[{gen_fsm,terminate,7,[{file,"gen_fsm.erl"},{line,600}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,239}]}]}
ancestors: [riak_core_vnode_sup,riak_core_sup,<0.162.0>]
messages: [{'$gen_sync_event',{<0.1289.0>,#Ref<0.0.0.4558>},wait_for_init}]
links: [<0.166.0>]
dictionary: [{random_seed,{19991,18664,29668}}]
trap_exit: true
status: running
heap_size: 987
stack_size: 27
reductions: 8154
I think the only thing is causing this problem is doing backups. Maybe docker container doesn't stop at all and Riak still operating on data. I've refractored backup script and for now everything seems to be ok.
By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.