Get libgcc dlls from opensuse as part of make win-extras

[tkelman]

This is to address JuliaInterop/ZMQ.jl#69

I originally wanted to make the win-extras target shorter, with an aim of getting rid of it as soon as we can expunge our dependency on command-line git for Pkg. Unfortunately putting these dll's in usr/bin breaks llvm-config.exe when compiled with Cygwin's version of MinGW-w64. Doing it as part of win-extras appears to work though, they get downloaded into dist-extras and only copied into the installed folder during make dist. No more need to restart Julia for dlopen of ZMQ or Cbc dll's to work.

This runs the risk of making the buildbots susceptible to opensuse connectivity flakiness, we'll have to see.

[tkelman]

@staticfloat have you ever seen �Error: undefined method []' for nil:NilClass` before from homebrew? https://s3.amazonaws.com/archive.travis-ci.org/jobs/51318570/log.txt

[staticfloat]

No, not quite sure what that's about. I've restarted the build, we'll see
what happens.
-E

On Wed, Feb 18, 2015 at 6:42 PM, Tony Kelman [email protected]
wrote:

@staticfloat https://github.com/staticfloat have you ever seen Error:
undefined method[]' for nil:NilClass` before from homebrew?
https://s3.amazonaws.com/archive.travis-ci.org/jobs/51318570/log.txt

—
Reply to this email directly or view it on GitHub
#10244 (comment).

[tkelman]

Fluke, apparently. Feedback on this approach, or should we just give it a shot on the buildbots and see what happens?

[staticfloat]

Yeah, I think we should just give it a shot

[tkelman]

BTW how do we control the buildbots staying up-to-date on their cygwin packages? IIRC packer/vagrant wound up not quite working with the type of virtualization you use for them?

[staticfloat]

I have an ansible script that I run every now and then to update everything
except the Windows builders. Do you have a command-line program that can
update cygwin packages?

On Thu, Feb 19, 2015 at 2:36 PM, Tony Kelman [email protected]
wrote:

BTW how do we control the buildbots staying up-to-date on their cygwin
packages? IIRC packer/vagrant wound up not quite working with the type of
virtualization you use for them?

—
Reply to this email directly or view it on GitHub
#10244 (comment).

[tkelman]

Yeah, same command that I use to initially install the cygwin packages, with the -g flag should upgrade installed packages. In powershell:

mkdir -Force C:\cygwin | Out-Null
(new-object net.webclient).DownloadFile(
  "http://cygwin.com/setup-x86_64.exe", "C:\cygwin\setup-x86_64.exe")
foreach ($pkg in @("git,make,curl,patch,python,gcc-g++,m4,cmake,p7zip",
    "mingw64-i686-gcc-g++,mingw64-i686-gcc-fortran",
    "mingw64-x86_64-gcc-g++,mingw64-x86_64-gcc-fortran")) {
  C:\cygwin\setup-x86_64.exe -q -n -R C:\cygwin -l C:\cygwin\packages `
    -s http://mirrors.chauf.net/cygwin -g -P $pkg | Where-Object `
    -FilterScript {$_ -notlike "Installing file *"} | Write-Output
}

[tkelman]

Looks like winrpm.sh works at least on the 64 bit buildbot http://buildbot.e.ip.saba.us:8010/builders/package_win8.1-x64/builds/329/steps/make%20win-extras/logs/stdio

The 32 bit build hit the same issue-10045 segfault that appveyor's been hitting.

We can look over the logs after running this for a few days and see whether or not the connectivity is a real problem. The opensuse build service rebuilds these rpm's automatically any time any of their dependencies change, and uploads a new revision number. I wonder whether the caching approach that we're using for homebrew on Travis could be made to work for these too...

[staticfloat]

Yes, it absolutely can. In fact, we can cache everything from winrpm if we
so desire. The caching code
https://github.com/staticfloat/cache.julialang.org/blob/master/www/cache.py#L57-L72
is really simple, right now it's set up to very rigidly only grab Homebrew
bottles, but we could expand it to be a whitelist instead of only that one
site. It would require a tiny bit of modification to our travis build, but
nothing major. (Since we'd need ot ask for cache.julialang.org/
instead of just cache.julialang.org/ like we do now.)
-E

On Thu, Feb 19, 2015 at 3:35 PM, Tony Kelman [email protected]
wrote:

Looks like winrpm.sh works at least on the 64 bit buildbot
http://buildbot.e.ip.saba.us:8010/builders/package_win8.1-x64/builds/329/steps/make%20win-extras/logs/stdio

The 32 bit build hit the same issue-10045 segfault that appveyor's been
hitting.

We can look over the logs after running this for a few days and see
whether or not the connectivity is a real problem. The opensuse build
service rebuilds these rpm's automatically any time any of their
dependencies change, and uploads a new revision number. I wonder whether
the caching approach that we're using for homebrew on Travis could be made
to work for these too...

—
Reply to this email directly or view it on GitHub
#10244 (comment).

[ihnorton]

The opensuse build service rebuilds these rpm's automatically any time any of their dependencies change

😭 because Windows testing is not enough Russian roulette already...

[tkelman]

Well, see the issue mentioned up top. And JuliaPackaging/WinRPM.jl#36. It would be great if the repository we were pulling binaries from was stable like a released linux distro, but it isn't, it's a continuously rebuilding development repo.

We should at least do the caching thing, that'll help with the reliability if not the actual upstream binary stability.

[staticfloat]

Alright, we just need to change the winrpm downloads to prepend
http://cache.e.ip.saba.us for us, and we'll be good to go on the caching
front.
-E

On Thu, Feb 19, 2015 at 3:46 PM, Tony Kelman [email protected]
wrote:

Well, see the issue mentioned up top. And JuliaPackaging/WinRPM.jl#36
JuliaPackaging/WinRPM.jl#36. It would be great if
the repository we were pulling binaries from was stable like a released
linux distro, but it isn't, it's a continuously rebuilding development repo.

We should at least do the caching thing, that'll help with the reliability
if not the actual upstream binary stability.

—
Reply to this email directly or view it on GitHub
#10244 (comment).

[tkelman]

Cool. We'll only want to do that for the rpm files, not the repodata xml since that is expected to change regularly.

[staticfloat]

Good call!

On Tue, Feb 24, 2015 at 2:43 AM, Tony Kelman [email protected]
wrote:

Cool. We'll only want to do that for the rpm files, not the repodata xml
since that is expected to change regularly.

—
Reply to this email directly or view it on GitHub
#10244 (comment).

[tkelman]

backported in cde7e4a and 484471c

I think the only thing that'll need to change for caching here is just switching the last instance of retry_curl (the only one that's downloading to a file instead of to a string) to use jldownload.