ex_unit: Add :capture_io tag

[wojtekmach]

Example:

defmodule MyTest do
  use ExUnit.Case, async: true
  @tag :capture_io
  test "with io", %{capture_io: io} do
    IO.puts("Hello, World!")
    assert StringIO.flush(io) == "Hello, World!\\n"
  end
end

---

Possible future work:

1. Allow ExUnit.start(capture_io: true) to mimic capture_log: true?

2. Add api to programmatically append to the StringIO input buffer, right now it's only possible on creation:

   {:ok, pid} = StringIO.open("3")
   "3" = IO.gets(pid, "1 + 2")

   The idea is something like this:

   {:ok, pid} = StringIO.open("")
   
   StringIO.buffer("1 + 2")
   "1 + 2" = IO.gets(pid, "> ")
   
   :eof = IO.gets(pid, "> ")
   
   StringIO.buffer("2 + 3")
   "2 + 3" = IO.gets(pid, "> ")

   And so we could replace the usage of:

   capture_io(prompt, fn -> ... end)

   too.

   Maybe instead of calling it StringIO.buffer it could be called StringIO.input_write and StringIO.input_puts to mimic IO.write and IO.puts. Not sure if we'd need .input_binwrite too.

3. @tag capture_io: :standard_error?

   I think we could do this as long as the test is running in async: false which we'd know and then we can raise otherwise. This is as opposed to capture_io(:standard_error, ...) where ExUnit does not know if it's sync or async.

   If we do this, do we want to capture both? I guess we could accept a list of devices and set it in context:

   @tag capture_io: [:stdio, :standard_error]
   test "foo", %{capture_io: [stdio, stderr]}

   Not sure if :stdio is even semantically correct, my understanding is it's not stdio per se but the group leader. So yeah, it's probably a pass, and a reason to use capture_io in this case, but I thought I'd mention this for completeness.

4. Make the StringIO available to @tag :capture_log tests in context as %{capture_log: io}. This could be considered a breaking change since today the context is usually set as %{capture_log: true} or %{capture_log: options} but I can't think of a reason to read them back in a test.

[sabiwara]

StringIO.buffer("1 + 2")

StringIO.buffer(pid, "1 + 2") right?

Make the StringIO available to @tag :capture_log tests in context as %{capture_log: io}. This could be considered a breaking change since today the context is usually set as %{capture_log: true} or %{capture_log: options} but I can't think of a reason to read them back in a test.

This one seems solvable by picking a slightly different name, captured_log: / captured_io:?

[sabiwara]

This is a really neat addition 🤩 💜

[josevalim]

@tag capture_io: :standard_error?

Right, this doesn't play well with StringIO.flush, because then you are flushing data between tests. I think it is fine to keep this specific to stdin/stdout. It is potentially the same reason why adding a similar API to capture_log would be confusing, you cannot use flush and that would lead to more imprecise tests. So I believe this is good to go as is.

[josevalim]

Maybe we should call it stdout for consistency with logs?

[wojtekmach]

Good call, test.stdout sounds good. Instead of context.capture_io should we have context.stdout too? I think that'd be nicer. And then same for logs, instead of context.capture_log (or per @sabiwara context.capture_logs) have context.logs?

[josevalim]

I think the issue is that we don't want to expose %{logs: device} because those devices are by definition shared. You will most likely read and flush data from other tests, so I think a tiny capture window works better.

In other words, we need to decide if we are happy shipping this feature considering it will only be available for stdout. The renaming to stdout in the struct should be a separate discussion.

[wojtekmach]

Are they shared though? I thought each test gets its own StringIO for capturing logs.

[josevalim]

Yes, you are right.

[josevalim]

The only thing holding this PR is to decide if we want to add a similar API to capture logs, such as:

@tag :capture_log
test "foo", %{capture_log: log} do
  assert StringIO.flush(log) =~ "..."
end

The issue is that we cannot add capture_log: log as that would be backwards incompatible. Therefore one option is to store the device under another name, such %{stdio: device, logs: device}.

---

I also realize that I lied. We could totally do capture_io: :stderr, as we give specific StringIO devices per process. So we also need an API for multiple devices. One option is to just place them as tags:

test "foo", %{captured_stdio: ..., captured_stderr, captured_logs: ...} do

But I am thinking some namespacing is probably best?

test "foo", %{devices: devices} do
  devices.stdio
  devices.stderr
  devices.logs
end

Possible names are devices or captured. Thoughts?

[wojtekmach]

test "foo", %{devices: devices} do
  devices.stdio
  devices.stderr
  devices.logs
end

I like this direction.

I wonder, would the full example be something like this?

@tag :capture_io # shortcut for capture_io: :stdout?
@tag capture_io: :stderr
@tag :capture_log
test "foo", %{devices: devices} do
  devices.stdio
  devices.stderr
  devices.logs
end

Or maybe explicit tag names:

@tag :capture_stdio
@tag :capture_stderr
@tag :capture_log
test "foo", %{devices: devices} do

Cause I'm not sure if we'd want to capture other devices in practice.

Or did you have something else in mind?

Could be namespaced on both input and output, capture/captured:

@tag capture: [:stdio]
test "foo", %{captured: %{stdio: io}}

@tag capture: [:stdio, :stderr, :logs]
test "foo", %{captured: %{stdio: _, stderr: _, logs: _}}

but I think @tag :capture_* reads way better.

I think :captured reads better than :devices, especially logs have their own devices I guess, but both are good imo.

[josevalim]

I think this reads well:

@tag :capture_stdio
@tag :capture_stderr
@tag :capture_log

They may receve options:

@tag capture_stdio: "input to be given, no need for buffer"
@tag :capture_stderr
@tag capture_log: :error

So not sure if we should do the nesting as we may end up with this: @tag capture: [:stderr, stdio: "buffer", log: :error]. But we could have @tag capture: :all... although I don't think I ever wanted to capture all three in practice.

---

Undecided between devices and captured. captured may read weird because nothing may been captured when we define the variable. It is all imperative.