2024-12-13
The following storage failure has been restored as follows:
(12/18: Points have been returned.)
- Date
Failure time : 2024-12-11 01:23
Recovery time : 2024-12-11 11:44
2. Impact and Details of the disorder
I had a storage failure and lost access to /home and /gs/fs. Due to this, login to TSUBAME became impossible. Also, TSUBAME portal was stopped around 10:20 to prevent secondary damage. There is a high possibility that jobs are failing during the outage.
3. Affected Jobs
The number of jobs that could have been affected by the outage (jobs that were running during the outage) was 591, and the number of reservations was 12.
=== Normal jobs : 591 ===
1791755 1791756 1792013 1792014 1796925 1796936 1796960 1832230 1834776 1834810 1844400 1859155 1859228 1859230 1859436 1867362 1867363 1867398 1867409 1867422 1867423 1867432 1867433 1867442 1867443 1867461 1867462 1867471 1867472 1867480 1867481 1867490 1867491 1867502 1867503 1867512 1867513 1867522 1867523 1867531 1867532 1867542 1867543 1867551 1867552 1867560 1867561 1867572 1867573 1867581 1867582 1870421 1870422 1870423 1870424 1870425 1870426 1870427 1870428 1870429 1870430 1870431 1870432 1870433 1870434 1870435 1870436 1870445 1870447 1870454 1870456 1870457 1870463 1870466 1870467 1870468 1870469 1870470 1870471 1870472 1870473 1878324 1878327 1878334 1894471 1894472 1894802 1895088 1895090 1895093 1895096 1895097 1895098 1895099 1895100 1895101 1895102 1895103 1897566 1899597 1899618 1900760 1900761 1900762 1900763 1900764 1900765 1900766 1900767 1900768 1900769 1900770 1900771 1900772 1900773 1900774 1900775 1900776 1900777 1900778 1900779 1900780 1900781 1900782 1900783 1900784 1900785 1900786 1900787 1900789 1900790 1901208 1901219 1901248 1902163 1902164 1902166 1902167 1902168 1902169 1902170 1902171 1902172 1902173 1902174 1902175 1902176 1902177 1902178 1902179 1902180 1902181 1902182 1902183 1902184 1902185 1902186 1902187 1902189 1902190 1902191 1902192 1902193 1902194 1902324 1902331 1902733 1902734 1902791 1902795 1902797 1902830 1903869 1903878 1903904 1903906 1904006 1904008 1904009 1904010 1904011 1904017 1904030 1904032 1904048 1904069 1904071 1904232 1904368 1904395 1904396 1904399 1904400 1904403 1904404 1904416 1904427 1904430 1904483 1904484 1904485 1904487 1904488 1904489 1904494 1904495 1904658 1904660 1904661 1904672 1904713 1904758 1904759 1904769 1904778 1904780 1904781 1904782 1904785 1904787 1904788 1904798 1904804 1904805 1904809 1904812 1904818 1904821 1904823 1904826 1904827 1904829 1904830 1904836 1904846 1904866 1904896 1904897 1904898 1904899 1904900 1904901 1904902 1904904 1904906 1904911 1904913 1904954 1904963 1904973 1904978 1904984 1905013 1905021 1905046 1905047 1905048 1905101 1905112 1905133 1905148 1905149 1905169 1905183 1905222 1905233 1905262 1905263 1905266 1905270 1905413 1905656 1905696 1905701 1905715 1905734 1905915 1906340 1906341 1906346 1906352 1906364 1906418 1906419 1906454 1906464 1906469 1906475 1906547 1906548 1906549 1906550 1906623 1906624 1906628 1906633 1906642 1906840 1906843 1906848 1906912 1906914 1906919 1906922 1906928 1906951 1906954 1906957 1906966 1907010 1907014 1907021 1907023 1907063 1907068 1907070 1907086 1907089 1907112 1907117 1907118 1907119 1907127 1907141 1907146 1907149 1907165 1907174 1907175 1907176 1907177 1907178 1907179 1907180 1907181 1907182 1907183 1907184 1907185 1907186 1907187 1907188 1907189 1907190 1907191 1907192 1907193 1907194 1907195 1907196 1907197 1907198 1907199 1907201 1907203 1907206 1907207 1907210 1907212 1907214 1907223 1907225 1907228 1907229 1907235 1907242 1907243 1907244 1907245 1907246 1907247 1907248 1907249 1907250 1907251 1907252 1907253 1907254 1907255 1907256 1907257 1907258 1907259 1907260 1907261 1907262 1907277 1907292 1907294 1907295 1907297 1907299 1907319 1907321 1907323 1907324 1907326 1907329 1907330 1907332 1907334 1907336 1907338 1907340 1907341 1907343 1907344 1907347 1907349 1907350 1907352 1907354 1907356 1907358 1907360 1907361 1907363 1907365 1907367 1907369 1907370 1907372 1907564 1907565 1907566 1907567 1907568 1907569 1907570 1907571 1907572 1907573 1907574 1907575 1907576 1907577 1907578 1907579 1907580 1907581 1907582 1907583 1907584 1907585 1907586 1907587 1907588 1907589 1907590 1907591 1907592 1907593 1907594 1907597 1907599 1907600 1907602 1907603 1907604 1907605 1907606 1907609 1907610 1907611 1907612 1907613 1907614 1907615 1907616 1907617 1907618 1907619 1907620 1907621 1907622 1907625 1907626 1907627 1907628 1907629 1907630 1907631 1907632 1907633 1907634 1907635 1907636 1907637 1907731 1908360 1908924 1908925 1908931 1908932 1908933 1908934 1908935 1908936 1908937 1908938 1908939 1908940 1908941 1908942 1908943 1908944 1908945 1908946 1908947 1908948 1908949 1908950 1908952 1908953 1908954 1908955 1908956 1908957 1908958 1908959 1908960 1908961 1908962 1908963 1908964 1908965 1908966 1908967 1908968 1908969 1908970 1908971 1908972 1908973 1908975 1908976 1908977 1908978 1908979 1908986 1908993 1909004 1909005 1909015 1909017 1909018 1909019 1909021 1909026 1909027 1909028 1909029 1909031 1909032 1909033 1909034 1909035 1909036 1909037 1909038 1909039 1909040 1909042 1909043 1909045 1909047 1909048 1909049 1909050 1909051 1910040 1910123 1910198 1910199 1910200 1910202 1910204 1910205 1910206 1910227 1910228 1910231 1910233 1910235 1910240 1910244 1910245 1910248 1910250 1910251 1910252
=== ar (Reservations)：12 ===
1815 1842 1876 1880 1888 1889 1890 1891 1897 1898 1899 1902
４．Prevention of recurrence and future actions
・(12/16) We will continue to investigate the cause of the problem.
・We are planning to return points for metered use and reserved jobs. (12/18) The return of points for jobs that may have been affected was completed around 16:00 on 12/17.