heasarc.query_region() gives different results

[jleagle94]

Hello,

I am trying to read in all CSC sources within some Galactic latitude range. I first start out with the following
sample = heasarc.query_region( SkyCoord(0, 0, unit='deg',frame='galactic'), spatial='all-sky', catalog='csc', maxrec=None )

and then I mask out sources that do not meet the Galactic latitude range I want. I have noticed that the query region above gives different results each time, despite having the same input.

I noticed it is always a sample of 1e5 sources with sample metrics that differ each time. Sometimes the minimum flux of a source in the sample is ~1e-10 and other times it will find ~1e-9, and it changes the results of my filtered sample. Sometimes I will get 71k sources with the latitude range I am exploring, and other times it will have 10k.

Am I doing something wrong that would make query_region() give different results each time? Should I use something else if I want to read in the entire catalog first like I am trying to do?

[bsipocz]

cc @zoghbi-a @d-giles

[zoghbi-a]

It looks to me like this is coming from how pyvo uses maxrec and the maximum limit set by the HEASARC service. Every time a query is made, a random subset of 1e5 entries is returned. It will need to be investigated further.

@jleagle94, for your case, you will need to use a TAP query directly to get the full table. Assuming you want the default set of columns, you can do:

# get a comma-separated list of the default columns in csc. Use columns = '*', to get all columns
columns = ', '.join(Heasarc._get_default_columns('csc'))
# construct a query for all entries; use TOP with a large number greater than the server's 1e5 LIMIT
query = f'SELECT TOP 9999999 {columns} FROM csc'
sample = Heasarc.query_tap(query).to_table()

This returned 407806 entries for me.

[jleagle94]

Thanks @zoghbi-a! This works great and I can confirm the sample does not change this way.

[bsipocz]

Do you want to add a note about it in the docs? Would that have helped you originally @jleagle94 ?

[jleagle94]

It definitely would have helped as I was wanting to pull all sources in the sky with no maximum record limit.

[bsipocz]

Would you be open for opening a pull request adding it to somewhere in docs/heasarc/heasarc.rst?