Possible memory leak involving warnings

[dkutner]

I recently updated my dplyr version (late to the party), and I'm hitting some increased memory usage. I've traced it back to how warnings are handled. Beginning in dplyr 1.1.1, I get the following output:

library(dplyr)
#> 
#> Attaching package: 'dplyr'
#> The following objects are masked from 'package:stats':
#> 
#>     filter, lag
#> The following objects are masked from 'package:base':
#> 
#>     intersect, setdiff, setequal, union

identity <- function(x, warn) {
    if (warn) {
        warning("fake warning")
    }
    x
}

df <- tibble::tibble(e = rep(1, 1e8))

print(gc())
#>             used  (Mb) gc trigger   (Mb)  max used  (Mb)
#> Ncells    620941  33.2    1306337   69.8   1306337  69.8
#> Vcells 101049658 771.0  148096356 1129.9 101084366 771.3

df <- df %>% mutate(e = identity(e, warn = TRUE))
#> Warning: There was 1 warning in `mutate()`.
#> ℹ In argument: `e = identity(e, warn = TRUE)`.
#> Caused by warning in `identity()`:
#> ! fake warning

print(gc())
#>             used  (Mb) gc trigger   (Mb)  max used  (Mb)
#> Ncells    729780  39.0    1306337   69.8   1306337  69.8
#> Vcells 101287706 772.8  148096356 1129.9 102369359 781.1

rm(df)
print(gc())
#>             used  (Mb) gc trigger   (Mb)  max used  (Mb)
#> Ncells    729742  39.0    1306337   69.8   1306337  69.8
#> Vcells 101287654 772.8  148096356 1129.9 102369359 781.1

^{Created on 2025-01-31 with reprex v2.0.2}

If I restart R and rerun with warn = FALSE, the final memory usage is only 7.9 MB rather than 772.8 MB. Additionally, if I rewrite the mutate to avoid using a pipe via df <- mutate(df, e = identity(e, warn = TRUE)), the final memory usage is only 8.8 MB. Switching the pipe to |> also yields low memory usage. Under dplyr 1.1.0, the above reprex yields 18.8 MB.

I don't have a full appreciation for whether warnings would capture my environment, but I'm wondering if that's perhaps happening within either base R or dplyr's own record of warnings.

[hadley]

I think you'll need to provide more compelling evidence that this is a problem:

library(dplyr, warn.conflicts = FALSE)

identity <- function(x, warn) {
    if (warn) {
        warning("fake warning")
    }
    x
}

df <- tibble::tibble(e = rep(1, 1e8))
old <- lobstr::mem_used()

df <- df %>% mutate(e = identity(e, warn = TRUE))
#> Warning: There was 1 warning in `mutate()`.
#> ℹ In argument: `e = identity(e, warn = TRUE)`.
#> Caused by warning in `identity()`:
#> ! fake warning
lobstr::mem_used() - old
#> 9.58 MB
old <- lobstr::mem_used()

df <- df %>% mutate(e = identity(e, warn = TRUE))
#> Warning: There was 1 warning in `mutate()`.
#> ℹ In argument: `e = identity(e, warn = TRUE)`.
#> Caused by warning in `identity()`:
#> ! fake warning
lobstr::mem_used() - old
#> 12.75 kB
old <- lobstr::mem_used()

df <- df %>% mutate(e = identity(e, warn = TRUE))
#> Warning: There was 1 warning in `mutate()`.
#> ℹ In argument: `e = identity(e, warn = TRUE)`.
#> Caused by warning in `identity()`:
#> ! fake warning
lobstr::mem_used() - old
#> 11.25 kB
old <- lobstr::mem_used()

df <- df %>% mutate(e = identity(e, warn = FALSE))
lobstr::mem_used() - old
#> 616 B
old <- lobstr::mem_used()

df <- df %>% mutate(e = identity(e, warn = FALSE))
lobstr::mem_used() - old
#> 632 B
old <- lobstr::mem_used()

df <- df %>% mutate(e = identity(e, warn = FALSE))
lobstr::mem_used() - old
#> 616 B
old <- lobstr::mem_used()

^{Created on 2025-02-18 with reprex v2.1.1}

[dkutner]

I likely misused the term "memory leak" since repeated applications don't lead to an increase in memory usage. I think the issue is that dplyr is holding onto a copy of whatever data frame last threw a warning. In my case outside of the reprex, dplyr is holding onto a 10 GB data frame and I run out of memory soon after. There's no way to release this memory, as far as I can tell, without suppressing warnings or throwing a new warning within another mutate.

library(dplyr, warn.conflicts = FALSE)

identity <- function(x, warn) {
    if (warn) {
        warning("fake warning")
    }
    x
}

df <- tibble::tibble(e = rep(1, 1e8))
old <- lobstr::mem_used()
rm(df)
lobstr::mem_used() - old
#> -800.00 MB

df <- tibble::tibble(e = rep(1, 1e8))
old <- lobstr::mem_used()
df <- df %>% mutate(e = identity(e, warn = TRUE))
#> Warning: There was 1 warning in `mutate()`.
#> ℹ In argument: `e = identity(e, warn = TRUE)`.
#> Caused by warning in `identity()`:
#> ! fake warning
rm(df)
lobstr::mem_used() - old
#> 7.99 MB

^{Created on 2025-02-18 with reprex v2.0.2}