kibble-scanners.git
2 months agoif we've scanned before, only grab latest changes master
Daniel Gruno [Wed, 19 Sep 2018 07:35:59 +0000 (09:35 +0200)] 
if we've scanned before, only grab latest changes

2 months agowe need to sleep to avoid abusing github
Daniel Gruno [Wed, 19 Sep 2018 07:19:57 +0000 (09:19 +0200)] 
we need to sleep to avoid abusing github

2 months agothis is extremely spammy when running manually :(
Daniel Gruno [Wed, 19 Sep 2018 07:12:51 +0000 (09:12 +0200)] 
this is extremely spammy when running manually :(

disable it for now

2 months agowe should just pass the response text..
Daniel Gruno [Wed, 19 Sep 2018 07:09:11 +0000 (09:09 +0200)] 
we should just pass the response text..

2 months agocould be no auth was set..
Daniel Gruno [Wed, 19 Sep 2018 07:01:59 +0000 (09:01 +0200)] 
could be no auth was set..

2 months agowe're only interested in label names for now
Daniel Gruno [Wed, 19 Sep 2018 06:58:02 +0000 (08:58 +0200)] 
we're only interested in label names for now

2 months agofix error logging
Daniel Gruno [Wed, 19 Sep 2018 06:55:17 +0000 (08:55 +0200)] 
fix error logging

wrong position of parens

2 months agowrong var name here
Daniel Gruno [Wed, 19 Sep 2018 06:39:35 +0000 (08:39 +0200)] 
wrong var name here

3 months agooops, typo
Daniel Gruno [Wed, 12 Sep 2018 11:40:52 +0000 (13:40 +0200)] 
oops, typo

3 months agoadd support for files changed as a list in each commit object
Daniel Gruno [Wed, 12 Sep 2018 10:16:09 +0000 (12:16 +0200)] 
add support for files changed as a list in each commit object

3 months agohandle closer key errors properly
Daniel Gruno [Sun, 9 Sep 2018 14:06:20 +0000 (16:06 +0200)] 
handle closer key errors properly

it's a dict...

3 months agocloser may be null for imported issues
Daniel Gruno [Sun, 9 Sep 2018 14:05:23 +0000 (16:05 +0200)] 
closer may be null for imported issues

3 months agobump default limits for tone/mood analysis
Daniel Gruno [Sun, 9 Sep 2018 14:03:50 +0000 (16:03 +0200)] 
bump default limits for tone/mood analysis

3 months agofix key errors
Daniel Gruno [Sun, 9 Sep 2018 14:03:18 +0000 (16:03 +0200)] 
fix key errors

sometimes the assignee/reporter is set but no email address is
available, so just try to get it non-fatally.

9 months agoprefer full name over username, if available
Daniel Gruno [Fri, 2 Mar 2018 18:51:32 +0000 (19:51 +0100)] 
prefer full name over username, if available

- only store shortened bio if it's new
- prefer full name over username if we find it.

9 months agoturns out this needs to be 0 for latest posts
Daniel Gruno [Fri, 2 Mar 2018 15:46:28 +0000 (16:46 +0100)] 
turns out this needs to be 0 for latest posts

9 months agoadd forum type, and a date-string field for histograms
Daniel Gruno [Fri, 2 Mar 2018 15:33:45 +0000 (16:33 +0100)] 
add forum type, and a date-string field for histograms

9 months agodon't kill the loop
Daniel Gruno [Fri, 2 Mar 2018 14:51:41 +0000 (15:51 +0100)] 
don't kill the loop

scope is wrong here. it should return True at the end,
if the loop is successful, and False within the loop when
an error occurs.

9 months agoWe can accept both version 1 and 2 atm
Daniel Gruno [Fri, 2 Mar 2018 14:46:53 +0000 (15:46 +0100)] 
We can accept both version 1 and 2 atm

...but 2 is obviously preferred!

9 months agoRewrite broker class, inherit kibble UI wrapper
Daniel Gruno [Fri, 2 Mar 2018 14:44:01 +0000 (15:44 +0100)] 
Rewrite broker class, inherit kibble UI wrapper

Instead of doing checks constantly, we'll inherit the
wrapper class from the UI repo. There are a few cases
where we still have to manually do if/else (bulk and
api checks), but the rest can be aliases.

Bump accepted DB version to 2.

9 months agoAlignm with new DB format
Daniel Gruno [Fri, 2 Mar 2018 11:07:45 +0000 (12:07 +0100)] 
Alignm with new DB format

if the DB is typeless, write to it and fetch from it
accordingly.

9 months agostore the person posting, d'uh.
Daniel Gruno [Mon, 26 Feb 2018 19:49:33 +0000 (20:49 +0100)] 
store the person posting, d'uh.

9 months agobe sure to actually store the post doc in ES
Daniel Gruno [Mon, 26 Feb 2018 19:47:54 +0000 (20:47 +0100)] 
be sure to actually store the post doc in ES

9 months agoremove spurious comment
Daniel Gruno [Mon, 26 Feb 2018 19:21:28 +0000 (20:21 +0100)] 
remove spurious comment

9 months agoUpdates to JSON API
Daniel Gruno [Mon, 26 Feb 2018 19:20:16 +0000 (20:20 +0100)] 
Updates to JSON API

- We want JSON, so specify that
- Sometimes we need a token (like for Travis) instead of basic auth

9 months agoAdd initial Discourse scanner plugin
Daniel Gruno [Mon, 26 Feb 2018 19:19:32 +0000 (20:19 +0100)] 
Add initial Discourse scanner plugin

9 months agoTry to grab the different states of 'started' jobs
Daniel Gruno [Fri, 23 Feb 2018 08:32:56 +0000 (09:32 +0100)] 
Try to grab the different states of 'started' jobs

hopefully I got this right and we'll be able to see the
diff between started and queued/blocked builds.

9 months agothis should use kibble's pprinter
Daniel Gruno [Wed, 21 Feb 2018 19:26:06 +0000 (20:26 +0100)] 
this should use kibble's pprinter

9 months agoneed to return true here
Daniel Gruno [Wed, 21 Feb 2018 19:24:53 +0000 (20:24 +0100)] 
need to return true here

this isn't a failed position to be in.

9 months agobreak if we hit the end
Daniel Gruno [Wed, 21 Feb 2018 19:23:12 +0000 (20:23 +0100)] 
break if we hit the end

continue is a bad choice, we need a clean break out of the loop

9 months agothe job URL needs to be consistent, tweak it
Daniel Gruno [Wed, 21 Feb 2018 18:37:45 +0000 (19:37 +0100)] 
the job URL needs to be consistent, tweak it

9 months agoreverse logic, fix a string
Daniel Gruno [Wed, 21 Feb 2018 18:29:58 +0000 (19:29 +0100)] 
reverse logic, fix a string

9 months agoscan all previous jobs, if that makes sense
Daniel Gruno [Wed, 21 Feb 2018 18:25:37 +0000 (19:25 +0100)] 
scan all previous jobs, if that makes sense

if we haven't scanned older jobs before, we scan them all.
This has some built-in logic that cancels a full scan
if we're on page 2 or above and find a build that we've seen
before, or if travis tells us to stop.

9 months agoqueue size should reflect multibuilds
Daniel Gruno [Wed, 21 Feb 2018 15:12:31 +0000 (16:12 +0100)] 
queue size should reflect multibuilds

some jobs have multiple builds, which all increase
the queue size (I'd think?). So let's multiply queue
size by number of concurrent jobs.

9 months agoAdd initial WIP Travis CI Scanner
Daniel Gruno [Wed, 21 Feb 2018 14:31:55 +0000 (15:31 +0100)] 
Add initial WIP Travis CI Scanner

H/T to Pono for the help here.

9 months agodon't bork if color isn't present
Daniel Gruno [Wed, 21 Feb 2018 09:48:25 +0000 (10:48 +0100)] 
don't bork if color isn't present

9 months agoallow for excluding scanner types with -e flag
Daniel Gruno [Wed, 21 Feb 2018 09:46:41 +0000 (10:46 +0100)] 
allow for excluding scanner types with -e flag

9 months agoalso count jobs building at the moment
Daniel Gruno [Mon, 19 Feb 2018 11:56:12 +0000 (12:56 +0100)] 
also count jobs building at the moment

9 months agoAdd in preliminary buildbot scanner
Daniel Gruno [Sat, 17 Feb 2018 09:29:55 +0000 (10:29 +0100)] 
Add in preliminary buildbot scanner

This isn't as advanced as the jenkins scanner (queues are
rather opaque in buildbot), but it'll show stuck builds,
as well as builds by duration/count.

9 months agowe need the jobURL, which is unique to a job
Daniel Gruno [Fri, 16 Feb 2018 18:25:32 +0000 (19:25 +0100)] 
we need the jobURL, which is unique to a job

This is so we can sum up build durations per job,
and sort them per CI (thus, if both Jenkins and Travis, for example, has
a job with the same name, we'll split them)

9 months agoadd a timestamp for when we think the build finished.
Daniel Gruno [Fri, 16 Feb 2018 18:18:17 +0000 (19:18 +0100)] 
add a timestamp for when we think the build finished.

9 months agoremove debug return
Daniel Gruno [Fri, 16 Feb 2018 18:16:10 +0000 (19:16 +0100)] 
remove debug return

we want this to actually scan :)

9 months agofix queue avg calc, add date key
Daniel Gruno [Fri, 16 Feb 2018 18:06:01 +0000 (19:06 +0100)] 
fix queue avg calc, add date key

avg needs to use max, not min.
add date, so we can do aggregations on dates

9 months agoAdd a generic Jenkins scanner
Daniel Gruno [Fri, 16 Feb 2018 12:13:35 +0000 (13:13 +0100)] 
Add a generic Jenkins scanner

This archives jobs done, queue size, avg wait time,
builds stuck/blocked and so on.

11 months agowe can't work on a jira ticket without fields data
Daniel Gruno [Mon, 15 Jan 2018 16:19:17 +0000 (17:19 +0100)] 
we can't work on a jira ticket without fields data

11 months agoIf not key phrases, put _NULL_ to avoid breaking ES
Daniel Gruno [Tue, 9 Jan 2018 01:37:27 +0000 (02:37 +0100)] 
If not key phrases, put _NULL_ to avoid breaking ES

ES does not seem to like empty sets here, so we'll
put _NULL_ in there when no phrases were found,
and ignore that in the UI.

11 months agobetter quote removal
Daniel Gruno [Tue, 9 Jan 2018 00:55:07 +0000 (01:55 +0100)] 
better quote removal

11 months agoBetter trimming of unnecessary text elements
Daniel Gruno [Tue, 9 Jan 2018 00:48:41 +0000 (01:48 +0100)] 
Better trimming of unnecessary text elements

We don't want to be analysing:
- quotes
- "on $date, bla bla wrote" sort of sentences
- URLs, email addresses

11 months agoforgot to add kpe to init.py
Daniel Gruno [Tue, 9 Jan 2018 00:32:31 +0000 (01:32 +0100)] 
forgot to add kpe to init.py

11 months agoInitial stab at KPE for Kibble
Daniel Gruno [Tue, 9 Jan 2018 00:29:09 +0000 (01:29 +0100)] 
Initial stab at KPE for Kibble

This only supports pony mail so far.
We'll have to work on support for Pipermail etc

11 months agoadditional emotional weighting available
Daniel Gruno [Fri, 29 Dec 2017 11:02:44 +0000 (12:02 +0100)] 
additional emotional weighting available

12 months agothere's a value for this too.
Daniel Gruno [Fri, 8 Dec 2017 18:39:14 +0000 (19:39 +0100)] 
there's a value for this too.

12 months agopicoAPI has scores for positivity/negativity, let's use those
Daniel Gruno [Fri, 8 Dec 2017 12:48:37 +0000 (13:48 +0100)] 
picoAPI has scores for positivity/negativity, let's use those

12 months agoweave picoAPI into pm-tone
Daniel Gruno [Fri, 8 Dec 2017 12:12:02 +0000 (13:12 +0100)] 
weave picoAPI into pm-tone

12 months agoadd support for picoAPI sentiment analysis
Daniel Gruno [Fri, 8 Dec 2017 12:11:05 +0000 (13:11 +0100)] 
add support for picoAPI sentiment analysis

The more the merrier

12 months agobump the limit from 1 email to 100 per scan at max
Daniel Gruno [Thu, 7 Dec 2017 10:36:10 +0000 (11:36 +0100)] 
bump the limit from 1 email to 100 per scan at max

12 months agoget ponymail-tone scanner to work with lib changes
Daniel Gruno [Thu, 7 Dec 2017 10:35:50 +0000 (11:35 +0100)] 
get ponymail-tone scanner to work with lib changes

grab all bodies, array them up, then scan them all at once

12 months agorework tone lib to accept an array of bodies
Daniel Gruno [Thu, 7 Dec 2017 10:35:24 +0000 (11:35 +0100)] 
rework tone lib to accept an array of bodies

Azure accepts up to 1000 bodies at the same time to
speed up and prevent rate limits, so let's make use of that.
also rework the watson to accept an array, even through
it's still one call per body.

12 months agoaccount for azure rate limiting
Daniel Gruno [Thu, 7 Dec 2017 10:13:59 +0000 (11:13 +0100)] 
account for azure rate limiting

12 months agoensure that azure returns a valid response
Daniel Gruno [Thu, 7 Dec 2017 09:58:15 +0000 (10:58 +0100)] 
ensure that azure returns a valid response

12 months agothis needs to be a string representation
Daniel Gruno [Thu, 7 Dec 2017 09:56:05 +0000 (10:56 +0100)] 
this needs to be a string representation

12 months agoadd commented out example watson/azure creds
Daniel Gruno [Wed, 6 Dec 2017 22:52:01 +0000 (23:52 +0100)] 
add commented out example watson/azure creds

12 months agoalso add azure text analysis option
Daniel Gruno [Wed, 6 Dec 2017 22:49:02 +0000 (23:49 +0100)] 
also add azure text analysis option

rename watson's to watsonTone.

12 months agocatch exception and store in db if we fail to scan
Daniel Gruno [Wed, 6 Dec 2017 11:13:25 +0000 (12:13 +0100)] 
catch exception and store in db if we fail to scan

12 months agoreport when we're done scanning
Daniel Gruno [Wed, 6 Dec 2017 11:12:02 +0000 (12:12 +0100)] 
report when we're done scanning

12 months ago^- (merghebegin work on a PoC twitter scanner
Daniel Gruno [Wed, 6 Dec 2017 11:11:02 +0000 (12:11 +0100)] 
^- (merghebegin work on a PoC twitter scanner

might be replaced with a streams container, but for now we'll
have something to work with, data-wise.

12 months agooverride alerts should cause a rewind attempt as well
Daniel Gruno [Mon, 27 Nov 2017 15:05:40 +0000 (16:05 +0100)] 
override alerts should cause a rewind attempt as well

When git says 'Your local changes to the following files would be
overwritten by checkout' we should probably try a rewind as well.

13 months agofail gracefully if watson breaks
Daniel Gruno [Tue, 24 Oct 2017 20:42:04 +0000 (22:42 +0200)] 
fail gracefully if watson breaks

let's do some better debug of this later on.

13 months agoneed to import the exceptions module
Daniel Gruno [Tue, 24 Oct 2017 17:55:14 +0000 (19:55 +0200)] 
need to import the exceptions module

13 months agocatch connection aborts
Daniel Gruno [Tue, 24 Oct 2017 17:48:33 +0000 (19:48 +0200)] 
catch connection aborts

13 months agothis is sometimes all caps, need to lowercase it all
Daniel Gruno [Tue, 24 Oct 2017 17:37:08 +0000 (19:37 +0200)] 
this is sometimes all caps, need to lowercase it all

13 months agobugzilla is also a known robit
Daniel Gruno [Tue, 24 Oct 2017 12:07:36 +0000 (14:07 +0200)] 
bugzilla is also a known robit

13 months agosome lists require credentials
Daniel Gruno [Tue, 24 Oct 2017 12:07:24 +0000 (14:07 +0200)] 
some lists require credentials

also switch to the json plugin for fetching data, since that
handles cookies already

13 months agocompact, add doc_as_upsert for updates which 404s
Daniel Gruno [Mon, 23 Oct 2017 21:47:43 +0000 (23:47 +0200)] 
compact, add doc_as_upsert for updates which 404s

13 months agoupsert isn't working
Daniel Gruno [Mon, 23 Oct 2017 21:39:08 +0000 (23:39 +0200)] 
upsert isn't working

gotta set the optype instead for bulks

13 months agorefactor logic
Daniel Gruno [Mon, 23 Oct 2017 21:26:19 +0000 (23:26 +0200)] 
refactor logic

- we want to stop after 50 emails
- we don't want robits
- emails already analysed count towards the 50!

13 months agograb a lot of email, bail at 50 legit ones
Daniel Gruno [Mon, 23 Oct 2017 18:42:45 +0000 (20:42 +0200)] 
grab a lot of email, bail at 50 legit ones

13 months agothese should be upserts
Daniel Gruno [Mon, 23 Oct 2017 18:38:24 +0000 (20:38 +0200)] 
these should be upserts

13 months agocrop out quotes from email
Daniel Gruno [Mon, 23 Oct 2017 17:54:18 +0000 (19:54 +0200)] 
crop out quotes from email

we're only interested in the actual reply, not what was quoted

13 months agoif watson borks, we bork silently
Daniel Gruno [Mon, 23 Oct 2017 17:47:19 +0000 (19:47 +0200)] 
if watson borks, we bork silently

13 months agoless confusion if we don't print the ignored emails
Daniel Gruno [Mon, 23 Oct 2017 17:09:08 +0000 (19:09 +0200)] 
less confusion if we don't print the ignored emails

let's only print debug data WHEN we find an email to analyse

13 months agoignore robits
Daniel Gruno [Mon, 23 Oct 2017 17:04:31 +0000 (19:04 +0200)] 
ignore robits

13 months agoadd ponymail tone analysis
Daniel Gruno [Mon, 23 Oct 2017 17:03:35 +0000 (19:03 +0200)] 
add ponymail tone analysis

Only enabled it watson configured.
set to only grab the last 50 emails, so as to not
abuse the service. WIP!

13 months agoadd a small plugin for tone analysis
Daniel Gruno [Mon, 23 Oct 2017 17:02:23 +0000 (19:02 +0200)] 
add a small plugin for tone analysis

13 months agokibblebit shortcut for dbname, notify when pushing stragglers
Daniel Gruno [Mon, 23 Oct 2017 17:02:02 +0000 (19:02 +0200)] 
kibblebit shortcut for dbname, notify when pushing stragglers

13 months agonote that ForkManager is required here
Daniel Gruno [Mon, 23 Oct 2017 10:20:48 +0000 (12:20 +0200)] 
note that ForkManager is required here

13 months agonote that a threaded cloc exists
Daniel Gruno [Mon, 23 Oct 2017 10:18:56 +0000 (12:18 +0200)] 
note that a threaded cloc exists

13 months agodefault to 'issue' issue type
Daniel Gruno [Sun, 22 Oct 2017 16:55:09 +0000 (18:55 +0200)] 
default to 'issue' issue type

This is so we can easily distinguish between pull/merge requests
and plain issues

13 months agouse kibblebit.append here, properly upsert docs
Daniel Gruno [Sun, 22 Oct 2017 16:27:28 +0000 (18:27 +0200)] 
use kibblebit.append here, properly upsert docs

13 months agokeep it quiet unless there is an error
Daniel Gruno [Sat, 21 Oct 2017 19:41:27 +0000 (21:41 +0200)] 
keep it quiet unless there is an error

git keeps 'odd' output for stderr, so pipe that
to stdout and only print if return code isn't 0

13 months agofix path, output brokenness
Daniel Gruno [Sat, 21 Oct 2017 19:27:42 +0000 (21:27 +0200)] 
fix path, output brokenness

- fix the path (cd to full path!)
- on error, spit out what went wrong

13 months agoonly display count on every 10th ticket
Daniel Gruno [Sat, 21 Oct 2017 19:09:39 +0000 (21:09 +0200)] 
only display count on every 10th ticket

we don't wanna flood when we have thousands of tickets.

13 months agoneeds to be forced UTC here
Daniel Gruno [Sat, 21 Oct 2017 09:59:38 +0000 (11:59 +0200)] 
needs to be forced UTC here

13 months agowe can safely push larger bulk objects
Daniel Gruno [Sat, 21 Oct 2017 09:44:10 +0000 (11:44 +0200)] 
we can safely push larger bulk objects

13 months agowe also accept trunk as a default, if no master
Daniel Gruno [Sat, 21 Oct 2017 09:40:09 +0000 (11:40 +0200)] 
we also accept trunk as a default, if no master

13 months agorename coc document
Daniel Gruno [Sat, 21 Oct 2017 09:19:32 +0000 (11:19 +0200)] 
rename coc document

13 months agostart working on a contributing doc
Daniel Gruno [Sat, 21 Oct 2017 09:18:17 +0000 (11:18 +0200)] 
start working on a contributing doc

13 months agoAdd the ASF Code of Conduct
Daniel Gruno [Sat, 21 Oct 2017 09:16:25 +0000 (11:16 +0200)] 
Add the ASF Code of Conduct

13 months agouse KibbleBit.pprint here
Daniel Gruno [Sat, 21 Oct 2017 08:53:32 +0000 (10:53 +0200)] 
use KibbleBit.pprint here

13 months agoupdate source status
Daniel Gruno [Sat, 21 Oct 2017 08:53:16 +0000 (10:53 +0200)] 
update source status