{"id":81,"date":"2020-09-01T22:44:29","date_gmt":"2020-09-02T03:44:29","guid":{"rendered":"https:\/\/www.lexneva.name\/blog\/?p=81"},"modified":"2020-09-01T22:57:42","modified_gmt":"2020-09-02T03:57:42","slug":"ad-0001","status":"publish","type":"post","link":"https:\/\/www.lexneva.name\/blog\/2020\/09\/01\/ad-0001\/","title":{"rendered":"AD 0001"},"content":{"rendered":"\n<p class=\"wp-block-paragraph\">The not-so-subtle irony of SRE Weekly is that <a href=\"https:\/\/sreweekly.com\/\">sreweekly.com<\/a> itself is really not very reliable at all[1].  It&#8217;s a little t3a.micro instance held together with duct tape and shell scripts.  There&#8217;s no monitoring.  It runs out of disk space constantly and I don&#8217;t find out until my email goes suspiciously quiet.  The only thing going for it is that it&#8217;s well-patched.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Last month, I had my <a href=\"https:\/\/twitter.com\/QuinnyPig\">@quinnypig<\/a> moment.  My AWS bill arrived and it was <strong>double<\/strong> what it usually is.  Crap!  Thank goodness twice a pittance is still a pittance, but I was curious, so I dug in a little.  Hilarity ensued.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">The reason for the high (for me) bill was immediately clear: the better part of a terabyte of data transfer out, plus some T3 Unlimited CPU credits.  The meager free Cloudwatch metrics showed that my instance had indeed pegged its CPU and data transfer out for a week, going back to normal just a bit over two weeks prior.<\/p>\n\n\n\n<ul class=\"wp-block-gallery columns-1 is-cropped wp-block-gallery-1 is-layout-flex wp-block-gallery-is-layout-flex\"><li class=\"blocks-gallery-item\"><figure><a href=\"https:\/\/www.lexneva.name\/blog\/wp-content\/uploads\/2020\/09\/aws_bill-1024x278.png\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"278\" src=\"https:\/\/www.lexneva.name\/blog\/wp-content\/uploads\/2020\/09\/aws_bill-1024x278.png\" alt=\"A partial screenshot of my AWS bill, with 687.839 GB of bandwidth and 107.692 vCPU-hours of T3A Unlimited credits highlighted.  The highlights are accompanied with an interrobang and double exclamation mark, respectively.\" data-id=\"84\" data-link=\"https:\/\/www.lexneva.name\/blog\/?attachment_id=84\" class=\"wp-image-84\" srcset=\"https:\/\/www.lexneva.name\/blog\/wp-content\/uploads\/2020\/09\/aws_bill-1024x278.png 1024w, https:\/\/www.lexneva.name\/blog\/wp-content\/uploads\/2020\/09\/aws_bill-300x81.png 300w, https:\/\/www.lexneva.name\/blog\/wp-content\/uploads\/2020\/09\/aws_bill-768x208.png 768w, https:\/\/www.lexneva.name\/blog\/wp-content\/uploads\/2020\/09\/aws_bill.png 1271w\" sizes=\"auto, (max-width: 1024px) 100vw, 1024px\" \/><\/a><\/figure><\/li><\/ul>\n\n\n\n<p class=\"wp-block-paragraph\">Aw crap.  Did I just get rooted?!  Was my little instance being used to mine bitcoins or serve porn?  Or mine porn?  Now stuff was getting serious.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">I quickly ran chkrootkit and it came up clean, not that that proves much.  I was promptly reminded that I only keep 2 weeks of Apache logs to save disk space.  No problem, I can just load up one of my EBS snapshot backups and look at its logs.  My backups.  The ones that totally exist.  They exist, right?!  Dammit.  My backup script had quietly started failing several months back.  Of course it had.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">I really had nothing to go on at this point.  Everything was back to normal, but who&#8217;s to say this wouldn&#8217;t happen again?  Maybe someone still had their tendrils hooked in and they were just waiting for me to resume my negligence?<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Okay, I decided, I guess it&#8217;s finally time for some monitoring.  I set up a few (free) Cloudwatch alerts on bandwidth and CPU usage and the like, shrugged, and reluctantly moved on.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Last Thursday, the alert fired.  Same exact symptoms.  I hopped in and checked the Apache log:<\/p>\n\n\n\n<pre class=\"wp-block-preformatted horizontal-scrolling-pre\">54.209.27.31 - lex [27\/Aug\/2020:06:48:39 +0000] \"POST \/tt-rss\/api\/ HTTP\/1.1\" 200 1648<br>54.209.27.31 &#8211; lex [27\/Aug\/2020:06:48:39 +0000] &#8220;POST \/tt-rss\/api\/ HTTP\/1.1&#8243; 200 545<\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">There it was, over and over again.  Tons of those.  Obviously someone&#8217;s trying to brute-force my RSS reader&#8217;s API!  And that IP resolves to an EC2 instance.  Awesome!  Case closed, add an iptables rule, beef up fail2ban, and start filling out an AWS abuse report.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Oh, they want to know what instance is being attacked, I should give them my IP address.<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">$ host sreweekly.com<br>sreweekly.com has address 54.209.27.31<\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">&#8230;&#8230;.. oh.  I see.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Thankfully, I hadn&#8217;t finished submitting that abuse report.  That could have been awkward.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Time to remove that iptables rule.  Now what the heck am I doing to myself?!  I had a sinking suspicion&#8230;<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">A little background:  I gather a lot of articles for the <a href=\"https:\/\/sreweekly.com\/\">newsletter<\/a> through a series of Google search alerts that I&#8217;ve fine-tuned over the years, and I subscribe to the alerts using my <a style=\"\" href=\"https:\/\/tt-rss.org\/\">RSS reader<\/a>.  <\/p>\n\n\n\n<p class=\"wp-block-paragraph\">The thing is, when something like Facebook goes down, everyone and their pet rock writes a news story about it.  They all have almost identical headlines.  I use a hacked up version of this nifty little script to sift out the duplicates:<\/p>\n\n\n\n<figure class=\"wp-block-image\"><a href=\"https:\/\/github.com\/reuteras\/newsdedup\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"300\" src=\"https:\/\/www.lexneva.name\/blog\/wp-content\/uploads\/2020\/09\/newsdedup-1024x300.png\" alt=\"This image is just a pointless screenshot of GitHub repo reuteras\/newsdedup, you can ignore it.\" class=\"wp-image-89\" srcset=\"https:\/\/www.lexneva.name\/blog\/wp-content\/uploads\/2020\/09\/newsdedup-1024x300.png 1024w, https:\/\/www.lexneva.name\/blog\/wp-content\/uploads\/2020\/09\/newsdedup-300x88.png 300w, https:\/\/www.lexneva.name\/blog\/wp-content\/uploads\/2020\/09\/newsdedup-768x225.png 768w, https:\/\/www.lexneva.name\/blog\/wp-content\/uploads\/2020\/09\/newsdedup.png 1052w\" sizes=\"auto, (max-width: 1024px) 100vw, 1024px\" \/><\/a><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\">It uses my RSS reader&#8217;s API to manage articles.  And sure enough, right there in the config file, it&#8217;s going out and coming back in through the front door, to make TLS easier:<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">[ttrss]<br>hostname=https:\/\/www.lexneva.name\/lex\/tt-rss<br><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">This little script has run smoothly for years, but now it was throwing exceptions.  A lot of them.  The main loop catches errors, prints them out, and tries again.  Fast.  It&#8217;s really good at it.  Trust me.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">I changed the code to print a full traceback (and sleep a couple seconds!) and started it back up.  Sure enough, it spewed:<\/p>\n\n\n\n<pre class=\"wp-block-preformatted horizontal-scrolling-pre\">Traceback (most recent call last):<br>   File &#8220;\/home\/lex\/repos\/newsdedup\/newsdedup.py&#8221;, line 220, in<br>     main()<br>   File &#8220;\/home\/lex\/repos\/newsdedup\/newsdedup.py&#8221;, line 215, in main<br>     learn_last_read(rss_api, title_queue, args, configuration)<br>   File &#8220;\/home\/lex\/repos\/newsdedup\/newsdedup.py&#8221;, line 72, in learn_last_read<br>     limit=limit, skip=seen)<br>   File &#8220;\/home\/lex\/.pyenv\/versions\/3.7.3\/lib\/python3.7\/site-packages\/ttrss\/client.py&#8221;, line 459, in headli$<br>     return self._client.get_headlines(feed_id=self.id, **kwargs)<br>   File &#8220;\/home\/lex\/.pyenv\/versions\/3.7.3\/lib\/python3.7\/site-packages\/ttrss\/client.py&#8221;, line 222, in get_headlines<br>     return [Headline(hl, self) for hl in r[&#8217;content&#8217;]]<br>   File &#8220;\/home\/lex\/.pyenv\/versions\/3.7.3\/lib\/python3.7\/site-packages\/ttrss\/client.py&#8221;, line 222, in <br>     return [Headline(hl, self) for hl in r[&#8217;content&#8217;]]<br>   File &#8220;\/home\/lex\/.pyenv\/versions\/3.7.3\/lib\/python3.7\/site-packages\/ttrss\/client.py&#8221;, line 481, in init<br>     self.updated = datetime.fromtimestamp(self.updated)<br> ValueError: year 0 is out of range<br><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Wat.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">I was fully expecting a bug in newsdedup.  I definitely wasn&#8217;t expecting that a simple ttrss API cliet function call would throw this kind of error.  WTF.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">I did a little googling for that error and a little digging in the code, and I started to wonder&#8230; just what kind of timestamp value (seconds since the epoch) would result in a year of 0?!  I asked the database:<\/p>\n\n\n\n<pre class=\"wp-block-preformatted horizontal-scrolling-pre\">mysql&gt; use ttrss;<br>Reading table information for completion of table and column names<br>You can turn off this feature to get a quicker startup with -A<br><br>Database changed<br>mysql&gt; select * from ttrss_entries order by updated asc limit 10 \\G<br><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">And there was my answer.  One of my feeds had 4 entries dated <strong>Mon, 01 Jan 0001 00:00:00 +0000<\/strong>.  AD 0001, stroke of midnight, literally the moment Jesus was conceived, or took his first breath, or whatever the heck, these blog posts were penned.  A Monday, as it turns out.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Some more background (I&#8217;m sorry it&#8217;ll be quick!): when I find one or two good articles by a given author, I often will drop their blog&#8217;s RSS feed into my reader in case they write anything good in the future.  Even if I only care about 1 in 10 things they write going forward, that&#8217;s still a <em>way<\/em> better signal-to-noise ratio than I get from Google search alerts.  And some of you folks are way higher than 10%, thank you!!<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">I&#8217;m guessing this AD 0001 date is a way of making a post appear way down at the bottom of the very last page in someone&#8217;s blog archive.  Totally reasonable.  I&#8217;d probably do the same.  And yet, my poor little script was barfing, and it wasn&#8217;t even something I could fix with my own code.  I&#8217;d have to fix the ttrss Python library.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">So to the person whose RSS feed triggered this: I&#8217;m sorry.  I had to unsubscribe.  It&#8217;s totally not your fault.  I think my Google alerts will probably catch anything new you write, but if not, definitely drop me a note.  Sorry!<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">And in the end, I&#8217;m still not sure where that &#8220;<a href=\"https:\/\/en.wikipedia.org\/wiki\/Anno_Domini#No_year_zero:_start_and_end_of_a_century\">year 0<\/a>&#8221; error came from.  The best I can figure is that it was trying to render the time in my local time zone (despite my server being in UTC!?), and since I&#8217;m over here in one of the negative time zones, it blew up.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">And in case you&#8217;re curious, the timestamp for 0001-01-01 00:00:00 UTC is -62135596800.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"> <br><br><\/p>\n\n\n\n<p class=\"has-small-font-size wp-block-paragraph\">[1] Please don&#8217;t DDoS sreweekly.com.  Please! It&#8217;s not funny and you&#8217;ll just make me sad.  <span style=\"color: red\">&#x2665;<\/span><\/p>\n","protected":false},"excerpt":{"rendered":"<p>The not-so-subtle irony of SRE Weekly is that sreweekly.com itself is really not very reliable at all[1]. It&#8217;s a little t3a.micro instance held together with duct tape and shell scripts. There&#8217;s no monitoring. It runs out of disk space constantly and I don&#8217;t find out until my email goes suspiciously quiet. The only thing going &bull;  <a class=\"read-more\" href=\"https:\/\/www.lexneva.name\/blog\/2020\/09\/01\/ad-0001\/\"> Read More &raquo;<\/a><\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"class_list":["post-81","post","type-post","status-publish","format-standard","hentry","category-sre"],"_links":{"self":[{"href":"https:\/\/www.lexneva.name\/blog\/wp-json\/wp\/v2\/posts\/81","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.lexneva.name\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.lexneva.name\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.lexneva.name\/blog\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.lexneva.name\/blog\/wp-json\/wp\/v2\/comments?post=81"}],"version-history":[{"count":37,"href":"https:\/\/www.lexneva.name\/blog\/wp-json\/wp\/v2\/posts\/81\/revisions"}],"predecessor-version":[{"id":123,"href":"https:\/\/www.lexneva.name\/blog\/wp-json\/wp\/v2\/posts\/81\/revisions\/123"}],"wp:attachment":[{"href":"https:\/\/www.lexneva.name\/blog\/wp-json\/wp\/v2\/media?parent=81"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.lexneva.name\/blog\/wp-json\/wp\/v2\/categories?post=81"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.lexneva.name\/blog\/wp-json\/wp\/v2\/tags?post=81"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}